[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222898807720-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:08:18 dist-test-slave-00ld krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:08:18 dist-test-slave-00ld krb5kdc[399](info): set up 2 sockets
Sep 24 04:08:18 dist-test-slave-00ld krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260924 04:08:19.885514 392 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222898807720-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222898807720-392-0/krb5kdc/test-user.keytab.
Sep 24 04:08:19 dist-test-slave-00ld krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:08:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:08:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:08:20.012342 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222898807720-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222898807720-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222898807720-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38595
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38595
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:08:20.123020 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:08:20.123351 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:08:20.123438 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:08:20.127439 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:08:20.127543 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:08:20.127581 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:08:20.127621 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:08:20.127655 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:08:20.132936 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222898807720-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222898807720-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38595
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38595
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222898807720-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:08:20.134028 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:08:20.135103 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:08:20.144287 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:08:20.144292 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:08:20.144522 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:08:20.144577 412 server_base.cc:1034] running on GCE node
I20260924 04:08:20.145153 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:08:20.146377 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:08:20.147538 412 hybrid_clock.cc:648] HybridClock initialized: now 1790222900147435 us; error 119 us; skew 500 ppm
Sep 24 04:08:20 dist-test-slave-00ld krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222900, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:08:20.150312 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:08:20.151320 412 webserver.cc:460] Webserver started at http://127.0.98.62:42741/ using document root <none> and password file <none>
I20260924 04:08:20.151898 412 fs_manager.cc:359] Metadata directory not provided
I20260924 04:08:20.151997 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:08:20.152240 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:08:20.154448 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222898807720-392-0/minicluster-data/master-0/data/instance:
uuid: "f1c677538b8149c88210f5d61b400e90"
format_stamp: "Formatted at 2026-09-24 04:08:20 on dist-test-slave-00ld"
server_key: "e281ad5d7bf89da4ec591bd8d3da29dd"
server_key_iv: "bc458288ef933fcddec5c47c8cbe5e70"
server_key_version: "encryptionkey@0"
I20260924 04:08:20.155035 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222898807720-392-0/minicluster-data/master-0/wal/instance:
uuid: "f1c677538b8149c88210f5d61b400e90"
format_stamp: "Formatted at 2026-09-24 04:08:20 on dist-test-slave-00ld"
server_key: "e281ad5d7bf89da4ec591bd8d3da29dd"
server_key_iv: "bc458288ef933fcddec5c47c8cbe5e70"
server_key_version: "encryptionkey@0"
I20260924 04:08:20.158908 412 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260924 04:08:20.161347 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:20.162606 412 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:08:20.162760 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222898807720-392-0/minicluster-data/master-0/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222898807720-392-0/minicluster-data/master-0/wal
uuid: "f1c677538b8149c88210f5d61b400e90"
format_stamp: "Formatted at 2026-09-24 04:08:20 on dist-test-slave-00ld"
server_key: "e281ad5d7bf89da4ec591bd8d3da29dd"
server_key_iv: "bc458288ef933fcddec5c47c8cbe5e70"
server_key_version: "encryptionkey@0"
I20260924 04:08:20.162878 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222898807720-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222898807720-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222898807720-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:08:20.193586 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:08:20.196707 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:08:20.196929 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:08:20.204097 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38595
I20260924 04:08:20.204123 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38595 every 8 connection(s)
I20260924 04:08:20.205063 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:08:20.207834 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:08:20.209116 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 412
I20260924 04:08:20.209239 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222898807720-392-0/minicluster-data/master-0/wal/instance
I20260924 04:08:20.210507 392 external_mini_cluster.cc:1428] Setting key c8ab877751d2b78ec67331f2f9f003f7
I20260924 04:08:20.214025 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1c677538b8149c88210f5d61b400e90: Bootstrap starting.
I20260924 04:08:20.217042 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f1c677538b8149c88210f5d61b400e90: Neither blocks nor log segments found. Creating new log.
I20260924 04:08:20.217967 480 log.cc:824] T 00000000000000000000000000000000 P f1c677538b8149c88210f5d61b400e90: Log is configured to *not* fsync() on all Append() calls
I20260924 04:08:20.220112 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1c677538b8149c88210f5d61b400e90: No bootstrap required, opened a new log
I20260924 04:08:20.224560 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P f1c677538b8149c88210f5d61b400e90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1c677538b8149c88210f5d61b400e90" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38595 } }
I20260924 04:08:20.224858 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P f1c677538b8149c88210f5d61b400e90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:08:20.224967 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P f1c677538b8149c88210f5d61b400e90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1c677538b8149c88210f5d61b400e90, State: Initialized, Role: FOLLOWER
I20260924 04:08:20.225459 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P f1c677538b8149c88210f5d61b400e90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1c677538b8149c88210f5d61b400e90" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38595 } }
I20260924 04:08:20.225634 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P f1c677538b8149c88210f5d61b400e90 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:08:20.225730 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P f1c677538b8149c88210f5d61b400e90 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:08:20.225853 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f1c677538b8149c88210f5d61b400e90 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:08:20.227000 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P f1c677538b8149c88210f5d61b400e90 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1c677538b8149c88210f5d61b400e90" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38595 } }
I20260924 04:08:20.227396 480 leader_election.cc:302] T 00000000000000000000000000000000 P f1c677538b8149c88210f5d61b400e90 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f1c677538b8149c88210f5d61b400e90; no voters:
I20260924 04:08:20.227746 480 leader_election.cc:288] T 00000000000000000000000000000000 P f1c677538b8149c88210f5d61b400e90 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:08:20.228042 485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f1c677538b8149c88210f5d61b400e90 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:08:20.228277 485 raft_consensus.cc:686] T 00000000000000000000000000000000 P f1c677538b8149c88210f5d61b400e90 [term 1 LEADER]: Becoming Leader. State: Replica: f1c677538b8149c88210f5d61b400e90, State: Running, Role: LEADER
I20260924 04:08:20.228631 485 consensus_queue.cc:237] T 00000000000000000000000000000000 P f1c677538b8149c88210f5d61b400e90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1c677538b8149c88210f5d61b400e90" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38595 } }
I20260924 04:08:20.229090 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P f1c677538b8149c88210f5d61b400e90 [sys.catalog]: configured and running, proceeding with master startup.
Sep 24 04:08:20 dist-test-slave-00ld krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:08:20.230284 487 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1c677538b8149c88210f5d61b400e90 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1c677538b8149c88210f5d61b400e90. Latest consensus state: current_term: 1 leader_uuid: "f1c677538b8149c88210f5d61b400e90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1c677538b8149c88210f5d61b400e90" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38595 } } }
I20260924 04:08:20.230409 487 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1c677538b8149c88210f5d61b400e90 [sys.catalog]: This master's current role is: LEADER
I20260924 04:08:20.231587 486 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1c677538b8149c88210f5d61b400e90 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f1c677538b8149c88210f5d61b400e90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1c677538b8149c88210f5d61b400e90" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38595 } } }
I20260924 04:08:20.231693 486 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1c677538b8149c88210f5d61b400e90 [sys.catalog]: This master's current role is: LEADER
I20260924 04:08:20.231920 494 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:08:20.234983 494 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:08:20.237218 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:20.213583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54936 (local address 127.0.98.62:38595)
0924 04:08:20.214047 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:20.214066 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:20.215086 (+ 1020us) server_negotiation.cc:407] Connection header received
0924 04:08:20.219263 (+ 4177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:20.219281 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:20.219602 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:20.219879 (+ 277us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:20.222876 (+ 2997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:20.223889 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:20.225582 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:20.226253 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:20.226378 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:20.231192 (+ 4814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:20.231197 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:20.231209 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:20.231234 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:20.233285 (+ 2051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:20.233910 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:20.233917 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:20.233922 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:20.234067 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:20.234462 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:20.234466 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:20.234468 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:20.234963 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:20.235157 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:20.235888 (+ 731us) server_negotiation.cc:299] Negotiation successful
0924 04:08:20.236193 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":136,"threads_started":1}
I20260924 04:08:20.240756 494 catalog_manager.cc:1347] Generated new cluster ID: 42fb31956a4d49449da7e4926f54f896
I20260924 04:08:20.240815 494 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:08:20.251793 494 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:08:20.252785 494 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:08:20.260732 494 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f1c677538b8149c88210f5d61b400e90: Generated new TSK 0
I20260924 04:08:20.261616 494 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222898807720-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222898807720-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:08:20.299238 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222898807720-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222898807720-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222898807720-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222898807720-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38595
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222898807720-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:08:20.411248 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:08:20.411553 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:08:20.411626 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:08:20.415624 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:08:20.415715 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:08:20.415812 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:08:20.420994 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222898807720-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222898807720-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222898807720-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222898807720-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38595
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222898807720-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:08:20.422041 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:08:20.423089 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:08:20.432776 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:08:20.432772 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:08:20.432776 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:08:20.432950 506 server_base.cc:1034] running on GCE node
I20260924 04:08:20.433454 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:08:20.434134 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:08:20.435289 506 hybrid_clock.cc:648] HybridClock initialized: now 1790222900435277 us; error 25 us; skew 500 ppm
Sep 24 04:08:20 dist-test-slave-00ld krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222900, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:08:20.438228 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:08:20.439073 506 webserver.cc:460] Webserver started at http://127.0.98.1:41013/ using document root <none> and password file <none>
I20260924 04:08:20.439599 506 fs_manager.cc:359] Metadata directory not provided
I20260924 04:08:20.439658 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:08:20.439819 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:08:20.441923 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222898807720-392-0/minicluster-data/ts-0/data/instance:
uuid: "a46bf5e94db947128549e295020bb75a"
format_stamp: "Formatted at 2026-09-24 04:08:20 on dist-test-slave-00ld"
server_key: "5830ed66f866e1b0c25926a2ba59a249"
server_key_iv: "654d629933d6908b1047d840815fe531"
server_key_version: "encryptionkey@0"
I20260924 04:08:20.442480 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222898807720-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a46bf5e94db947128549e295020bb75a"
format_stamp: "Formatted at 2026-09-24 04:08:20 on dist-test-slave-00ld"
server_key: "5830ed66f866e1b0c25926a2ba59a249"
server_key_iv: "654d629933d6908b1047d840815fe531"
server_key_version: "encryptionkey@0"
I20260924 04:08:20.446280 506 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 04:08:20.448809 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:20.450099 506 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:08:20.450230 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222898807720-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222898807720-392-0/minicluster-data/ts-0/wal
uuid: "a46bf5e94db947128549e295020bb75a"
format_stamp: "Formatted at 2026-09-24 04:08:20 on dist-test-slave-00ld"
server_key: "5830ed66f866e1b0c25926a2ba59a249"
server_key_iv: "654d629933d6908b1047d840815fe531"
server_key_version: "encryptionkey@0"
I20260924 04:08:20.450353 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222898807720-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222898807720-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222898807720-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:08:20.463335 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:08:20.466651 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:08:20.466894 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:08:20.467608 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:08:20.468473 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:08:20.468545 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:20.468614 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:08:20.468662 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:20.478755 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43383
I20260924 04:08:20.478776 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43383 every 8 connection(s)
I20260924 04:08:20.479664 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222898807720-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:08:20.485750 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 506
I20260924 04:08:20.485841 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222898807720-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:08:20.486126 392 external_mini_cluster.cc:1428] Setting key 721ac74cd24ccb9ae8730c8890738863
Sep 24 04:08:20 dist-test-slave-00ld krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222900, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:08:20.493747 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:20.481495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42279 (local address 127.0.98.62:38595)
0924 04:08:20.481661 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:20.481664 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:20.482509 (+ 845us) server_negotiation.cc:407] Connection header received
0924 04:08:20.483883 (+ 1374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:20.483886 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:20.483943 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:20.484022 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:20.485257 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:20.485868 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:20.487043 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:20.487297 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:20.487415 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:20.489970 (+ 2555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:20.489973 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:20.489976 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:20.490010 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:20.491574 (+ 1564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:20.492124 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:20.492129 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:20.492130 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:20.492215 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:20.492557 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:20.492562 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:20.492564 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:20.492817 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:20.492976 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:20.493461 (+ 485us) server_negotiation.cc:299] Negotiation successful
0924 04:08:20.493586 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 04:08:20.494547 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:20.481895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38595 (local address 127.0.98.1:42279)
0924 04:08:20.482333 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:20.482389 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:20.483662 (+ 1273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:20.484189 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:20.484196 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:20.484555 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:08:20.485112 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:20.485126 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:20.485982 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:20.485986 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:20.486916 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:20.486925 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:20.487451 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:20.487455 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:20.487689 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:20.488308 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:08:20.488325 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:08:20.489804 (+ 1479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:08:20.491713 (+ 1909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:20.491718 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:20.491734 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:20.492004 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:20.492315 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:20.492318 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:20.492320 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:20.492463 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:20.492969 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:08:20.492975 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:08:20.493225 (+ 250us) client_negotiation.cc:769] Sending connection context
0924 04:08:20.493424 (+ 199us) client_negotiation.cc:240] Negotiation successful
0924 04:08:20.493744 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":112,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 04:08:20.495712 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38595
I20260924 04:08:20.496030 635 heartbeater.cc:464] Registering TS with master...
I20260924 04:08:20.496469 635 heartbeater.cc:511] Master 127.0.98.62:38595 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 04:08:20.498164 444 ts_manager.cc:194] Registered new tserver with Master: a46bf5e94db947128549e295020bb75a (127.0.98.1:43383)
I20260924 04:08:20.499480 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:42279
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222898807720-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222898807720-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:08:20.515331 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222898807720-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222898807720-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222898807720-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222898807720-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38595
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222898807720-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:08:20.625363 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:08:20.625896 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:08:20.626020 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:08:20.630280 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:08:20.630332 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:08:20.630424 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:08:20.636286 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222898807720-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222898807720-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222898807720-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222898807720-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38595
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222898807720-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:08:20.637365 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:08:20.638363 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:08:20.648876 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:08:20.648881 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:08:20.648904 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:08:20.649189 640 server_base.cc:1034] running on GCE node
I20260924 04:08:20.649650 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:08:20.650362 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:08:20.651557 640 hybrid_clock.cc:648] HybridClock initialized: now 1790222900651526 us; error 58 us; skew 500 ppm
Sep 24 04:08:20 dist-test-slave-00ld krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222900, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:08:20.654520 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:08:20.655514 640 webserver.cc:460] Webserver started at http://127.0.98.2:36587/ using document root <none> and password file <none>
I20260924 04:08:20.656108 640 fs_manager.cc:359] Metadata directory not provided
I20260924 04:08:20.656199 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:08:20.656435 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:08:20.658581 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222898807720-392-0/minicluster-data/ts-1/data/instance:
uuid: "2979128ba5b74a3aac7343f97fde5ea7"
format_stamp: "Formatted at 2026-09-24 04:08:20 on dist-test-slave-00ld"
server_key: "056dee12470989afa28a9e87d26a83d5"
server_key_iv: "1b9d725caab8ce73b66b45ba74535f75"
server_key_version: "encryptionkey@0"
I20260924 04:08:20.659173 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222898807720-392-0/minicluster-data/ts-1/wal/instance:
uuid: "2979128ba5b74a3aac7343f97fde5ea7"
format_stamp: "Formatted at 2026-09-24 04:08:20 on dist-test-slave-00ld"
server_key: "056dee12470989afa28a9e87d26a83d5"
server_key_iv: "1b9d725caab8ce73b66b45ba74535f75"
server_key_version: "encryptionkey@0"
I20260924 04:08:20.663127 640 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 04:08:20.665812 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:20.667315 640 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:08:20.667465 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222898807720-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222898807720-392-0/minicluster-data/ts-1/wal
uuid: "2979128ba5b74a3aac7343f97fde5ea7"
format_stamp: "Formatted at 2026-09-24 04:08:20 on dist-test-slave-00ld"
server_key: "056dee12470989afa28a9e87d26a83d5"
server_key_iv: "1b9d725caab8ce73b66b45ba74535f75"
server_key_version: "encryptionkey@0"
I20260924 04:08:20.667586 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222898807720-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222898807720-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222898807720-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:08:20.679430 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:08:20.682717 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:08:20.682924 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:08:20.683503 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:08:20.684312 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:08:20.684396 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:20.684471 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:08:20.684515 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:20.695403 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40299
I20260924 04:08:20.695472 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40299 every 8 connection(s)
I20260924 04:08:20.696374 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222898807720-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:08:20.701562 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 640
I20260924 04:08:20.701697 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222898807720-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:08:20.701943 392 external_mini_cluster.cc:1428] Setting key 2f47c4386d23a38588a0b4adf840a9ff
Sep 24 04:08:20 dist-test-slave-00ld krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222900, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 04:08:20.711735 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:20.698379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:32787 (local address 127.0.98.62:38595)
0924 04:08:20.698524 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:20.698529 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:20.699585 (+ 1056us) server_negotiation.cc:407] Connection header received
0924 04:08:20.700948 (+ 1363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:20.700951 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:20.700995 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:20.701070 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:20.702346 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:20.702994 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:20.704243 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:20.704637 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:20.704798 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:20.708158 (+ 3360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:20.708161 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:20.708163 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:20.708187 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:20.709661 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:20.710163 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:20.710168 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:20.710169 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:20.710251 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:20.710589 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:20.710594 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:20.710595 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:20.710771 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:20.710892 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:20.711448 (+ 556us) server_negotiation.cc:299] Negotiation successful
0924 04:08:20.711566 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 04:08:20.712514 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:20.698817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38595 (local address 127.0.98.2:32787)
0924 04:08:20.699432 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:20.699474 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:20.700668 (+ 1194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:20.701228 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:20.701236 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:20.701604 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:08:20.702197 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:20.702213 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:20.703130 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:20.703133 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:20.704117 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:20.704128 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:20.704850 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:20.704856 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:20.705152 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:20.705779 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:08:20.705809 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:08:20.707989 (+ 2180us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:08:20.709784 (+ 1795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:20.709790 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:20.709803 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:20.710061 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:20.710358 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:20.710361 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:20.710363 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:20.710498 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:20.710890 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:08:20.710897 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:08:20.711139 (+ 242us) client_negotiation.cc:769] Sending connection context
0924 04:08:20.711385 (+ 246us) client_negotiation.cc:240] Negotiation successful
0924 04:08:20.711703 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":108,"threads_started":1}
I20260924 04:08:20.713681 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38595
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260924 04:08:20.713939 769 heartbeater.cc:464] Registering TS with master...
I20260924 04:08:20.714445 769 heartbeater.cc:511] Master 127.0.98.62:38595 requested a full tablet report, sending...
I20260924 04:08:20.715534 444 ts_manager.cc:194] Registered new tserver with Master: 2979128ba5b74a3aac7343f97fde5ea7 (127.0.98.2:40299)
I20260924 04:08:20.716426 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:32787
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222898807720-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222898807720-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:08:20.729909 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222898807720-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222898807720-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222898807720-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222898807720-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38595
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222898807720-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:08:20.842984 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:08:20.843263 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:08:20.843307 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:08:20.847486 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:08:20.847551 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:08:20.847642 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:08:20.853025 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222898807720-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222898807720-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222898807720-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222898807720-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38595
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222898807720-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:08:20.854143 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:08:20.855172 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:08:20.864940 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:08:20.865029 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:08:20.864950 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:08:20.865120 774 server_base.cc:1034] running on GCE node
I20260924 04:08:20.865655 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:08:20.866421 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:08:20.867575 774 hybrid_clock.cc:648] HybridClock initialized: now 1790222900867562 us; error 26 us; skew 500 ppm
Sep 24 04:08:20 dist-test-slave-00ld krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222900, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:08:20.870400 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:08:20.871488 774 webserver.cc:460] Webserver started at http://127.0.98.3:42429/ using document root <none> and password file <none>
I20260924 04:08:20.872128 774 fs_manager.cc:359] Metadata directory not provided
I20260924 04:08:20.872217 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:08:20.872457 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:08:20.874620 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222898807720-392-0/minicluster-data/ts-2/data/instance:
uuid: "dd484ca501bb4b5c8ff79414b7d3ed4b"
format_stamp: "Formatted at 2026-09-24 04:08:20 on dist-test-slave-00ld"
server_key: "40d0d1fd6a3f2ac245a4d0c1aa9e6bee"
server_key_iv: "df6958416a77f7814ea88b350e7cbf66"
server_key_version: "encryptionkey@0"
I20260924 04:08:20.875200 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222898807720-392-0/minicluster-data/ts-2/wal/instance:
uuid: "dd484ca501bb4b5c8ff79414b7d3ed4b"
format_stamp: "Formatted at 2026-09-24 04:08:20 on dist-test-slave-00ld"
server_key: "40d0d1fd6a3f2ac245a4d0c1aa9e6bee"
server_key_iv: "df6958416a77f7814ea88b350e7cbf66"
server_key_version: "encryptionkey@0"
I20260924 04:08:20.879047 774 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 04:08:20.881539 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:20.882938 774 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:08:20.883092 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222898807720-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222898807720-392-0/minicluster-data/ts-2/wal
uuid: "dd484ca501bb4b5c8ff79414b7d3ed4b"
format_stamp: "Formatted at 2026-09-24 04:08:20 on dist-test-slave-00ld"
server_key: "40d0d1fd6a3f2ac245a4d0c1aa9e6bee"
server_key_iv: "df6958416a77f7814ea88b350e7cbf66"
server_key_version: "encryptionkey@0"
I20260924 04:08:20.883211 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222898807720-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222898807720-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222898807720-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:08:20.894266 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:08:20.897768 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:08:20.897976 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:08:20.898577 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:08:20.899389 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:08:20.899461 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:20.899533 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:08:20.899576 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:20.909556 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44277
I20260924 04:08:20.909575 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44277 every 8 connection(s)
I20260924 04:08:20.910497 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222898807720-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:08:20.916451 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 774
I20260924 04:08:20.916544 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790222898807720-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:08:20.916831 392 external_mini_cluster.cc:1428] Setting key 6afafbd7401500e86f8efaeb80b441c4
Sep 24 04:08:20 dist-test-slave-00ld krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222900, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:08:20.924449 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:20.912472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50435 (local address 127.0.98.62:38595)
0924 04:08:20.912611 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:20.912615 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:20.913440 (+ 825us) server_negotiation.cc:407] Connection header received
0924 04:08:20.914863 (+ 1423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:20.914867 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:20.914929 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:20.915031 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:20.916187 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:20.916849 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:20.918227 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:20.918488 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:20.918586 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:20.921157 (+ 2571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:20.921160 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:20.921162 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:20.921185 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:20.922627 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:20.923134 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:20.923137 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:20.923138 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:20.923211 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:20.923515 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:20.923517 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:20.923518 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:20.923686 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:20.923760 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:20.924231 (+ 471us) server_negotiation.cc:299] Negotiation successful
0924 04:08:20.924323 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 04:08:20.925316 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:20.912839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38595 (local address 127.0.98.3:50435)
0924 04:08:20.913268 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:20.913321 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:20.914619 (+ 1298us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:20.915205 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:20.915212 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:20.915565 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:08:20.916034 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:20.916047 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:20.916971 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:20.916975 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:20.918098 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:20.918107 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:20.918624 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:20.918627 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:20.918772 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:20.919475 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:08:20.919496 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:08:20.920997 (+ 1501us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:08:20.922740 (+ 1743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:20.922745 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:20.922756 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:20.923044 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:20.923291 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:20.923293 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:20.923294 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:20.923422 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:20.923774 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:08:20.923779 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:08:20.924029 (+ 250us) client_negotiation.cc:769] Sending connection context
0924 04:08:20.924186 (+ 157us) client_negotiation.cc:240] Negotiation successful
0924 04:08:20.924439 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":111,"threads_started":1}
I20260924 04:08:20.926380 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38595
I20260924 04:08:20.926626 903 heartbeater.cc:464] Registering TS with master...
I20260924 04:08:20.927105 903 heartbeater.cc:511] Master 127.0.98.62:38595 requested a full tablet report, sending...
I20260924 04:08:20.928066 444 ts_manager.cc:194] Registered new tserver with Master: dd484ca501bb4b5c8ff79414b7d3ed4b (127.0.98.3:44277)
I20260924 04:08:20.928645 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50435
I20260924 04:08:20.931409 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 04:08:20 dist-test-slave-00ld krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222900, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 04:08:20 dist-test-slave-00ld krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222900, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 04:08:20.957187 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:20.947538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55470 (local address 127.0.98.1:43383)
0924 04:08:20.947880 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:20.947916 (+ 36us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:20.947940 (+ 24us) server_negotiation.cc:407] Connection header received
0924 04:08:20.948008 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:20.948013 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:20.948201 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:20.948311 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:20.949030 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:20.949787 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:20.950646 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:20.951069 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:20.951164 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:20.953630 (+ 2466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:20.953638 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:20.953653 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:20.953676 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:20.955469 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:20.955848 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:20.955852 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:20.955853 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:20.955957 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:20.956181 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:20.956183 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:20.956184 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:20.956413 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:20.956559 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:20.956772 (+ 213us) server_negotiation.cc:299] Negotiation successful
0924 04:08:20.956969 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":156,"threads_started":1}
W20260924 04:08:20.958209 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55470
Sep 24 04:08:20 dist-test-slave-00ld krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222900, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 04:08:20 dist-test-slave-00ld krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222900, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 04:08:20.984843 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:20.975952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55482 (local address 127.0.98.1:43383)
0924 04:08:20.976133 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:20.976137 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:20.976609 (+ 472us) server_negotiation.cc:407] Connection header received
0924 04:08:20.976773 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:20.976776 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:20.976824 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:20.976897 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:20.977632 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:20.978195 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:20.978993 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:20.979214 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:20.979292 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:20.981732 (+ 2440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:20.981735 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:20.981737 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:20.981762 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:20.983114 (+ 1352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:20.983533 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:20.983535 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:20.983536 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:20.983606 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:20.983929 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:20.983931 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:20.983933 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:20.984087 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:20.984181 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:20.984477 (+ 296us) server_negotiation.cc:299] Negotiation successful
0924 04:08:20.984575 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260924 04:08:20.985913 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 506
I20260924 04:08:20.991966 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 640
I20260924 04:08:20.999646 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 774
I20260924 04:08:21.005252 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 412
2026-09-24T04:08:21Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2191 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222898807720-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:08:21 dist-test-slave-00ld krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:08:21 dist-test-slave-00ld krb5kdc[917](info): set up 2 sockets
Sep 24 04:08:21 dist-test-slave-00ld krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260924 04:08:22.106757 392 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260924 04:08:22.209470 392 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.103s user 0.000s sys 0.002s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222898807720-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222898807720-392-0/krb5kdc/test-user.keytab.
Sep 24 04:08:22 dist-test-slave-00ld krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222902, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:08:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:08:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:08:22.375831 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222898807720-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222898807720-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222898807720-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41957
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41957
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:08:22.487583 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:08:22.487859 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:08:22.487900 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:08:22.491956 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:08:22.492014 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:08:22.492033 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:08:22.492050 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:08:22.492067 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:08:22.497347 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222898807720-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222898807720-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41957
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222898807720-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:08:22.498334 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:08:22.499291 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:08:22.507822 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:08:22.507844 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:08:22.507858 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:08:22.508006 930 server_base.cc:1034] running on GCE node
I20260924 04:08:22.508791 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:08:22.509877 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:08:22.511049 930 hybrid_clock.cc:648] HybridClock initialized: now 1790222902511036 us; error 27 us; skew 500 ppm
Sep 24 04:08:22 dist-test-slave-00ld krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222902, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:08:22.513819 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:08:22.514655 930 webserver.cc:460] Webserver started at http://127.0.98.62:39267/ using document root <none> and password file <none>
I20260924 04:08:22.515187 930 fs_manager.cc:359] Metadata directory not provided
I20260924 04:08:22.515244 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:08:22.515406 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:08:22.517436 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222898807720-392-0/minicluster-data/master-0/data/instance:
uuid: "94d08c20f14c48a9a3b49232b588a148"
format_stamp: "Formatted at 2026-09-24 04:08:22 on dist-test-slave-00ld"
server_key: "ac7d7d43d320933263068a02cb11925c"
server_key_iv: "ab63b7fb93f3080ffce2779719184fe8"
server_key_version: "encryptionkey@0"
I20260924 04:08:22.517926 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222898807720-392-0/minicluster-data/master-0/wal/instance:
uuid: "94d08c20f14c48a9a3b49232b588a148"
format_stamp: "Formatted at 2026-09-24 04:08:22 on dist-test-slave-00ld"
server_key: "ac7d7d43d320933263068a02cb11925c"
server_key_iv: "ab63b7fb93f3080ffce2779719184fe8"
server_key_version: "encryptionkey@0"
I20260924 04:08:22.521502 930 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:08:22.523739 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:22.525038 930 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:08:22.525169 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222898807720-392-0/minicluster-data/master-0/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222898807720-392-0/minicluster-data/master-0/wal
uuid: "94d08c20f14c48a9a3b49232b588a148"
format_stamp: "Formatted at 2026-09-24 04:08:22 on dist-test-slave-00ld"
server_key: "ac7d7d43d320933263068a02cb11925c"
server_key_iv: "ab63b7fb93f3080ffce2779719184fe8"
server_key_version: "encryptionkey@0"
I20260924 04:08:22.525272 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222898807720-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222898807720-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222898807720-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:08:22.551029 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:08:22.554149 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:08:22.554350 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:08:22.561424 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41957
I20260924 04:08:22.561450 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41957 every 8 connection(s)
I20260924 04:08:22.562381 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:08:22.565169 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:08:22.570587 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94d08c20f14c48a9a3b49232b588a148: Bootstrap starting.
I20260924 04:08:22.571973 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 930
I20260924 04:08:22.572114 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222898807720-392-0/minicluster-data/master-0/wal/instance
I20260924 04:08:22.572384 392 external_mini_cluster.cc:1428] Setting key 86575769f90ab918492ca028e13bb876
I20260924 04:08:22.573232 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 94d08c20f14c48a9a3b49232b588a148: Neither blocks nor log segments found. Creating new log.
I20260924 04:08:22.573977 998 log.cc:824] T 00000000000000000000000000000000 P 94d08c20f14c48a9a3b49232b588a148: Log is configured to *not* fsync() on all Append() calls
I20260924 04:08:22.576254 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94d08c20f14c48a9a3b49232b588a148: No bootstrap required, opened a new log
I20260924 04:08:22.581008 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 94d08c20f14c48a9a3b49232b588a148 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94d08c20f14c48a9a3b49232b588a148" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41957 } }
I20260924 04:08:22.581329 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 94d08c20f14c48a9a3b49232b588a148 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:08:22.581420 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 94d08c20f14c48a9a3b49232b588a148 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94d08c20f14c48a9a3b49232b588a148, State: Initialized, Role: FOLLOWER
I20260924 04:08:22.581979 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 94d08c20f14c48a9a3b49232b588a148 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94d08c20f14c48a9a3b49232b588a148" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41957 } }
I20260924 04:08:22.582152 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 94d08c20f14c48a9a3b49232b588a148 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
Sep 24 04:08:22 dist-test-slave-00ld krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222902, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:08:22.582239 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 94d08c20f14c48a9a3b49232b588a148 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:08:22.582347 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 94d08c20f14c48a9a3b49232b588a148 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:08:22.583406 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 94d08c20f14c48a9a3b49232b588a148 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94d08c20f14c48a9a3b49232b588a148" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41957 } }
I20260924 04:08:22.583768 998 leader_election.cc:302] T 00000000000000000000000000000000 P 94d08c20f14c48a9a3b49232b588a148 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 94d08c20f14c48a9a3b49232b588a148; no voters:
I20260924 04:08:22.584101 998 leader_election.cc:288] T 00000000000000000000000000000000 P 94d08c20f14c48a9a3b49232b588a148 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:08:22.584210 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 94d08c20f14c48a9a3b49232b588a148 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:08:22.584462 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P 94d08c20f14c48a9a3b49232b588a148 [term 1 LEADER]: Becoming Leader. State: Replica: 94d08c20f14c48a9a3b49232b588a148, State: Running, Role: LEADER
I20260924 04:08:22.584807 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P 94d08c20f14c48a9a3b49232b588a148 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94d08c20f14c48a9a3b49232b588a148" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41957 } }
I20260924 04:08:22.585614 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 94d08c20f14c48a9a3b49232b588a148 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:08:22.586422 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94d08c20f14c48a9a3b49232b588a148 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "94d08c20f14c48a9a3b49232b588a148" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94d08c20f14c48a9a3b49232b588a148" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41957 } } }
I20260924 04:08:22.586539 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94d08c20f14c48a9a3b49232b588a148 [sys.catalog]: This master's current role is: LEADER
I20260924 04:08:22.586874 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94d08c20f14c48a9a3b49232b588a148 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94d08c20f14c48a9a3b49232b588a148. Latest consensus state: current_term: 1 leader_uuid: "94d08c20f14c48a9a3b49232b588a148" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94d08c20f14c48a9a3b49232b588a148" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41957 } } }
I20260924 04:08:22.586977 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94d08c20f14c48a9a3b49232b588a148 [sys.catalog]: This master's current role is: LEADER
I20260924 04:08:22.588603 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:22.573569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43772 (local address 127.0.98.62:41957)
0924 04:08:22.574084 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:22.574121 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:22.574178 (+ 57us) server_negotiation.cc:407] Connection header received
0924 04:08:22.574994 (+ 816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:22.575013 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:22.575357 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:22.575638 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:22.576634 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:22.577687 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:22.579041 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:22.579513 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:22.579638 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:22.582944 (+ 3306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:22.582949 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:22.582962 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:22.582986 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:22.585169 (+ 2183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:22.585651 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:22.585656 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:22.585662 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:22.585823 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:22.586153 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:22.586158 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:22.586160 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:22.586710 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:22.586933 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:22.587361 (+ 428us) server_negotiation.cc:299] Negotiation successful
0924 04:08:22.587673 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":165,"threads_started":1}
W20260924 04:08:22.590497 1018 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 94d08c20f14c48a9a3b49232b588a148: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:08:22.590585 1018 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:08:22.590660 1011 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:08:22.591426 1011 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:08:22.596736 1011 catalog_manager.cc:1347] Generated new cluster ID: b4fd101a7004489caf497843e982fd5a
I20260924 04:08:22.596802 1011 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:08:22.604571 1011 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:08:22.605521 1011 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:08:22.610466 1011 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 94d08c20f14c48a9a3b49232b588a148: Generated new TSK 0
I20260924 04:08:22.611129 1011 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222898807720-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222898807720-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:08:22.648181 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222898807720-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222898807720-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222898807720-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222898807720-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41957
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222898807720-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:08:22.784900 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:08:22.785142 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:08:22.785187 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:08:22.789285 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:08:22.789346 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:08:22.789405 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:08:22.794473 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222898807720-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222898807720-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222898807720-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222898807720-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41957
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222898807720-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:08:22.795487 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:08:22.796434 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:08:22.805794 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:08:22.805799 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:08:22.806102 1024 server_base.cc:1034] running on GCE node
W20260924 04:08:22.806120 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:08:22.806550 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:08:22.807192 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:08:22.808384 1024 hybrid_clock.cc:648] HybridClock initialized: now 1790222902808368 us; error 28 us; skew 500 ppm
Sep 24 04:08:22 dist-test-slave-00ld krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222902, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:08:22.811142 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:08:22.811965 1024 webserver.cc:460] Webserver started at http://127.0.98.1:46047/ using document root <none> and password file <none>
I20260924 04:08:22.812511 1024 fs_manager.cc:359] Metadata directory not provided
I20260924 04:08:22.812598 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:08:22.812853 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:08:22.814813 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222898807720-392-0/minicluster-data/ts-0/data/instance:
uuid: "277c3fe31eca45a983c87b1df46dd0f7"
format_stamp: "Formatted at 2026-09-24 04:08:22 on dist-test-slave-00ld"
server_key: "f96afb52b113bcf924e45975fa665c5b"
server_key_iv: "184602b0d8942673328195107ab398f2"
server_key_version: "encryptionkey@0"
I20260924 04:08:22.815373 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222898807720-392-0/minicluster-data/ts-0/wal/instance:
uuid: "277c3fe31eca45a983c87b1df46dd0f7"
format_stamp: "Formatted at 2026-09-24 04:08:22 on dist-test-slave-00ld"
server_key: "f96afb52b113bcf924e45975fa665c5b"
server_key_iv: "184602b0d8942673328195107ab398f2"
server_key_version: "encryptionkey@0"
I20260924 04:08:22.818991 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 04:08:22.821339 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:22.822530 1024 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:08:22.822675 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222898807720-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222898807720-392-0/minicluster-data/ts-0/wal
uuid: "277c3fe31eca45a983c87b1df46dd0f7"
format_stamp: "Formatted at 2026-09-24 04:08:22 on dist-test-slave-00ld"
server_key: "f96afb52b113bcf924e45975fa665c5b"
server_key_iv: "184602b0d8942673328195107ab398f2"
server_key_version: "encryptionkey@0"
I20260924 04:08:22.822784 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222898807720-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222898807720-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222898807720-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:08:22.852051 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:08:22.855518 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:08:22.855732 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:08:22.856364 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:08:22.857302 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:08:22.857349 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:22.857421 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:08:22.857453 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:22.865868 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43681
I20260924 04:08:22.865916 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43681 every 8 connection(s)
I20260924 04:08:22.867046 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222898807720-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:08:22.870046 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 1024
I20260924 04:08:22.870158 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222898807720-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:08:22.870434 392 external_mini_cluster.cc:1428] Setting key d340d1789b3996d30ece735fd04c7671
Sep 24 04:08:22 dist-test-slave-00ld krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222902, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 04:08:22.882278 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:22.868995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57961 (local address 127.0.98.62:41957)
0924 04:08:22.869128 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:22.869133 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:22.871022 (+ 1889us) server_negotiation.cc:407] Connection header received
0924 04:08:22.872509 (+ 1487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:22.872512 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:22.872562 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:22.872654 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:22.873840 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:22.874373 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:22.875513 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:22.875749 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:22.875862 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:22.878628 (+ 2766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:22.878631 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:22.878633 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:22.878667 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:22.880090 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:22.880707 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:22.880712 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:22.880714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:22.880796 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:22.881158 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:22.881161 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:22.881163 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:22.881359 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:22.881468 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:22.881977 (+ 509us) server_negotiation.cc:299] Negotiation successful
0924 04:08:22.882099 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 04:08:22.883169 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:22.869465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41957 (local address 127.0.98.1:57961)
0924 04:08:22.870871 (+ 1406us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:22.870914 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:22.872285 (+ 1371us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:22.872872 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:22.872879 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:22.873231 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:08:22.873690 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:22.873702 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:22.874497 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:22.874500 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:22.875385 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:22.875393 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:22.875895 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:22.875899 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:22.876068 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:22.876795 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:08:22.876812 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:08:22.878456 (+ 1644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:08:22.880210 (+ 1754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:22.880216 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:22.880232 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:22.880587 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:22.880894 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:22.880898 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:22.880900 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:22.881053 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:22.881465 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:08:22.881472 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:08:22.881728 (+ 256us) client_negotiation.cc:769] Sending connection context
0924 04:08:22.881927 (+ 199us) client_negotiation.cc:240] Negotiation successful
0924 04:08:22.882264 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":142,"threads_started":1}
I20260924 04:08:22.884455 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41957
I20260924 04:08:22.884756 1153 heartbeater.cc:464] Registering TS with master...
I20260924 04:08:22.885336 1153 heartbeater.cc:511] Master 127.0.98.62:41957 requested a full tablet report, sending...
I20260924 04:08:22.887151 962 ts_manager.cc:194] Registered new tserver with Master: 277c3fe31eca45a983c87b1df46dd0f7 (127.0.98.1:43681)
I20260924 04:08:22.888343 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57961
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222898807720-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222898807720-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:08:22.900358 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222898807720-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222898807720-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222898807720-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222898807720-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41957
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222898807720-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:08:23.008245 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:08:23.008550 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:08:23.008628 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:08:23.012635 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:08:23.012741 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:08:23.012873 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:08:23.017993 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222898807720-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222898807720-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222898807720-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222898807720-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41957
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222898807720-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:08:23.018985 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:08:23.019966 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:08:23.029491 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:08:23.029491 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:08:23.029556 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:08:23.029722 1158 server_base.cc:1034] running on GCE node
I20260924 04:08:23.030242 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:08:23.030898 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:08:23.032101 1158 hybrid_clock.cc:648] HybridClock initialized: now 1790222903032077 us; error 39 us; skew 500 ppm
Sep 24 04:08:23 dist-test-slave-00ld krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222903, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:08:23.034956 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:08:23.035836 1158 webserver.cc:460] Webserver started at http://127.0.98.2:34087/ using document root <none> and password file <none>
I20260924 04:08:23.036369 1158 fs_manager.cc:359] Metadata directory not provided
I20260924 04:08:23.036461 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:08:23.036667 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:08:23.038655 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222898807720-392-0/minicluster-data/ts-1/data/instance:
uuid: "f2c8bf16eb554b0daf7a1854b87cffcb"
format_stamp: "Formatted at 2026-09-24 04:08:23 on dist-test-slave-00ld"
server_key: "651c77c4caf098ca523d2b2e9ea7bc02"
server_key_iv: "69b64f8222774571961deb1a588e40aa"
server_key_version: "encryptionkey@0"
I20260924 04:08:23.039213 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222898807720-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f2c8bf16eb554b0daf7a1854b87cffcb"
format_stamp: "Formatted at 2026-09-24 04:08:23 on dist-test-slave-00ld"
server_key: "651c77c4caf098ca523d2b2e9ea7bc02"
server_key_iv: "69b64f8222774571961deb1a588e40aa"
server_key_version: "encryptionkey@0"
I20260924 04:08:23.043058 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.005s
I20260924 04:08:23.045487 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:23.046768 1158 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 04:08:23.046912 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222898807720-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222898807720-392-0/minicluster-data/ts-1/wal
uuid: "f2c8bf16eb554b0daf7a1854b87cffcb"
format_stamp: "Formatted at 2026-09-24 04:08:23 on dist-test-slave-00ld"
server_key: "651c77c4caf098ca523d2b2e9ea7bc02"
server_key_iv: "69b64f8222774571961deb1a588e40aa"
server_key_version: "encryptionkey@0"
I20260924 04:08:23.047044 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222898807720-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222898807720-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222898807720-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:08:23.062609 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:08:23.065761 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:08:23.065968 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:08:23.066555 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:08:23.067363 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:08:23.067431 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:23.067500 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:08:23.067548 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:23.076318 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38305
I20260924 04:08:23.076385 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38305 every 8 connection(s)
I20260924 04:08:23.077320 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222898807720-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:08:23.086473 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 1158
Sep 24 04:08:23 dist-test-slave-00ld krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222903, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:08:23.086597 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222898807720-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:08:23.086899 392 external_mini_cluster.cc:1428] Setting key 4f365deee0dab2e078170104b48d9628
I20260924 04:08:23.091652 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:23.079073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38425 (local address 127.0.98.62:41957)
0924 04:08:23.079227 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:23.079230 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:23.079995 (+ 765us) server_negotiation.cc:407] Connection header received
0924 04:08:23.081324 (+ 1329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:23.081326 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:23.081371 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:23.081446 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:23.082578 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:23.083111 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:23.084217 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:23.084450 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:23.084539 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:23.087629 (+ 3090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:23.087633 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:23.087636 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:23.087708 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:23.089664 (+ 1956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:23.090165 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:23.090170 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:23.090172 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:23.090255 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:23.090589 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:23.090594 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:23.090595 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:23.090786 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:23.090893 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:23.091376 (+ 483us) server_negotiation.cc:299] Negotiation successful
0924 04:08:23.091499 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 04:08:23.092512 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:23.079414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41957 (local address 127.0.98.2:38425)
0924 04:08:23.079838 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:23.079879 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:23.081125 (+ 1246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:23.081612 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:23.081618 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:23.081962 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:08:23.082457 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:23.082470 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:23.083220 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:23.083222 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:23.084076 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:23.084084 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:23.084582 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:23.084585 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:23.084786 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:23.085386 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:08:23.085402 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:08:23.087176 (+ 1774us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:08:23.089783 (+ 2607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:23.089788 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:23.089799 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:23.090049 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:23.090355 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:23.090358 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:23.090360 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:23.090493 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:23.090887 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:08:23.090893 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:08:23.091142 (+ 249us) client_negotiation.cc:769] Sending connection context
0924 04:08:23.091338 (+ 196us) client_negotiation.cc:240] Negotiation successful
0924 04:08:23.091649 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":110,"threads_started":1}
I20260924 04:08:23.093637 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41957
I20260924 04:08:23.093883 1287 heartbeater.cc:464] Registering TS with master...
I20260924 04:08:23.094352 1287 heartbeater.cc:511] Master 127.0.98.62:41957 requested a full tablet report, sending...
I20260924 04:08:23.095388 962 ts_manager.cc:194] Registered new tserver with Master: f2c8bf16eb554b0daf7a1854b87cffcb (127.0.98.2:38305)
I20260924 04:08:23.096217 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:38425
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222898807720-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222898807720-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:08:23.116302 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222898807720-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222898807720-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222898807720-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222898807720-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41957
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222898807720-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:08:23.223680 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:08:23.223918 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:08:23.223963 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:08:23.228034 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:08:23.228087 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:08:23.228147 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:08:23.233196 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222898807720-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222898807720-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222898807720-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222898807720-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41957
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222898807720-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:08:23.234112 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:08:23.235023 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:08:23.244428 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:08:23.244438 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:08:23.244438 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:08:23.244870 1292 server_base.cc:1034] running on GCE node
I20260924 04:08:23.245226 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:08:23.245884 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:08:23.247051 1292 hybrid_clock.cc:648] HybridClock initialized: now 1790222903247032 us; error 32 us; skew 500 ppm
Sep 24 04:08:23 dist-test-slave-00ld krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222903, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:08:23.250274 1292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:08:23.251381 1292 webserver.cc:460] Webserver started at http://127.0.98.3:46545/ using document root <none> and password file <none>
I20260924 04:08:23.251981 1292 fs_manager.cc:359] Metadata directory not provided
I20260924 04:08:23.252043 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:08:23.252270 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:08:23.254487 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222898807720-392-0/minicluster-data/ts-2/data/instance:
uuid: "02113ec861ca4024a5bf8c09c2728c9f"
format_stamp: "Formatted at 2026-09-24 04:08:23 on dist-test-slave-00ld"
server_key: "7623579639ecdcabf948263bc523236a"
server_key_iv: "7071ca3311adc9cd2a9d9b7bc4ffbb74"
server_key_version: "encryptionkey@0"
I20260924 04:08:23.255070 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222898807720-392-0/minicluster-data/ts-2/wal/instance:
uuid: "02113ec861ca4024a5bf8c09c2728c9f"
format_stamp: "Formatted at 2026-09-24 04:08:23 on dist-test-slave-00ld"
server_key: "7623579639ecdcabf948263bc523236a"
server_key_iv: "7071ca3311adc9cd2a9d9b7bc4ffbb74"
server_key_version: "encryptionkey@0"
I20260924 04:08:23.258952 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 04:08:23.261556 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:23.262778 1292 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:08:23.262909 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222898807720-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222898807720-392-0/minicluster-data/ts-2/wal
uuid: "02113ec861ca4024a5bf8c09c2728c9f"
format_stamp: "Formatted at 2026-09-24 04:08:23 on dist-test-slave-00ld"
server_key: "7623579639ecdcabf948263bc523236a"
server_key_iv: "7071ca3311adc9cd2a9d9b7bc4ffbb74"
server_key_version: "encryptionkey@0"
I20260924 04:08:23.263018 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222898807720-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222898807720-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222898807720-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:08:23.278931 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:08:23.282145 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:08:23.282349 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:08:23.282922 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:08:23.283715 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:08:23.283761 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:23.283834 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:08:23.283874 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:23.293092 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46317
I20260924 04:08:23.293125 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46317 every 8 connection(s)
I20260924 04:08:23.293978 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222898807720-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:08:23.302106 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 1292
I20260924 04:08:23.302256 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790222898807720-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:08:23.302515 392 external_mini_cluster.cc:1428] Setting key 5c097dbc13c6f681d3620c11ef090940
Sep 24 04:08:23 dist-test-slave-00ld krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222903, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:08:23.308182 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:23.295687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34609 (local address 127.0.98.62:41957)
0924 04:08:23.295864 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:23.295867 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:23.296637 (+ 770us) server_negotiation.cc:407] Connection header received
0924 04:08:23.298065 (+ 1428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:23.298068 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:23.298125 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:23.298225 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:23.299362 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:23.299886 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:23.301728 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:23.301951 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:23.302062 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:23.304855 (+ 2793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:23.304858 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:23.304860 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:23.304883 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:23.306343 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:23.306813 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:23.306815 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:23.306816 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:23.306886 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:23.307261 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:23.307263 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:23.307264 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:23.307423 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:23.307493 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:23.307978 (+ 485us) server_negotiation.cc:299] Negotiation successful
0924 04:08:23.308068 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 04:08:23.309062 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:23.296062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41957 (local address 127.0.98.3:34609)
0924 04:08:23.296478 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:23.296519 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:23.297851 (+ 1332us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:23.298399 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:23.298406 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:23.298755 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:08:23.299216 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:23.299229 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:23.300007 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:23.300011 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:23.301580 (+ 1569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:23.301590 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:23.302108 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:23.302112 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:23.302349 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:23.303004 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:08:23.303024 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:08:23.304635 (+ 1611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:08:23.306442 (+ 1807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:23.306447 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:23.306458 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:23.306701 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:23.306993 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:23.306998 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:23.307000 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:23.307166 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:23.307505 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:08:23.307510 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:08:23.307729 (+ 219us) client_negotiation.cc:769] Sending connection context
0924 04:08:23.307914 (+ 185us) client_negotiation.cc:240] Negotiation successful
0924 04:08:23.308153 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":118,"threads_started":1}
I20260924 04:08:23.310130 1421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41957
I20260924 04:08:23.310374 1421 heartbeater.cc:464] Registering TS with master...
I20260924 04:08:23.310840 1421 heartbeater.cc:511] Master 127.0.98.62:41957 requested a full tablet report, sending...
I20260924 04:08:23.311780 962 ts_manager.cc:194] Registered new tserver with Master: 02113ec861ca4024a5bf8c09c2728c9f (127.0.98.3:46317)
I20260924 04:08:23.312336 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34609
I20260924 04:08:23.316407 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:08:23.327452 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:23.319853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43776 (local address 127.0.98.62:41957)
0924 04:08:23.319975 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:23.319978 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:23.320151 (+ 173us) server_negotiation.cc:407] Connection header received
0924 04:08:23.320256 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:23.320258 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:23.320294 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:23.320360 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:23.321131 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:23.321646 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:23.322448 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:23.322649 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:23.322730 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:23.324031 (+ 1301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:23.324034 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:23.324046 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:23.324068 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:23.325619 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:23.326076 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:23.326081 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:23.326083 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:23.326193 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:23.326544 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:23.326548 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:23.326550 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:23.326763 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:23.326903 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:23.327143 (+ 240us) server_negotiation.cc:299] Negotiation successful
0924 04:08:23.327270 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 04:08:23.332302 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43776:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:08:23.335319 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:08:23.350611 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:23.344424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38305 (local address 127.0.0.1:53532)
0924 04:08:23.344844 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:23.344894 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:23.345024 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:23.346458 (+ 1434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:23.346464 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:23.346507 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:08:23.346658 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:23.346669 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:23.347750 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:23.347753 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:23.349157 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:23.349166 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:23.349998 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:23.350001 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:23.350111 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:23.350158 (+ 47us) client_negotiation.cc:769] Sending connection context
0924 04:08:23.350320 (+ 162us) client_negotiation.cc:240] Negotiation successful
0924 04:08:23.350450 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":93,"threads_started":1}
I20260924 04:08:23.350621 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:23.344936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53532 (local address 127.0.98.2:38305)
0924 04:08:23.345935 (+ 999us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:23.345949 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:23.345977 (+ 28us) server_negotiation.cc:407] Connection header received
0924 04:08:23.346064 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:23.346069 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:23.346301 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:23.346454 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:08:23.346788 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:23.347601 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:23.349265 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:23.349860 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:23.349988 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:23.350146 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:23.350350 (+ 204us) server_negotiation.cc:299] Negotiation successful
0924 04:08:23.350450 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":865,"thread_start_us":108,"threads_started":1}
I20260924 04:08:23.351540 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:23.345395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43681 (local address 127.0.0.1:37204)
0924 04:08:23.346086 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:23.346100 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:23.346182 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:23.347196 (+ 1014us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:23.347199 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:23.347224 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:08:23.347321 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:23.347327 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:23.348812 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:23.348815 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:23.349961 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:23.349970 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:23.351086 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:23.351089 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:23.351189 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:23.351212 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 04:08:23.351291 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 04:08:23.351413 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":633,"mutex_wait_us":1,"thread_start_us":94,"threads_started":1}
I20260924 04:08:23.351565 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:23.345654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37204 (local address 127.0.98.1:43681)
0924 04:08:23.346841 (+ 1187us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:23.346853 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:23.346870 (+ 17us) server_negotiation.cc:407] Connection header received
0924 04:08:23.346926 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:23.346930 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:23.347069 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:23.347220 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:08:23.347448 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:23.348639 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:23.350094 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:23.350961 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:23.351066 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:23.351198 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:23.351324 (+ 126us) server_negotiation.cc:299] Negotiation successful
0924 04:08:23.351411 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1081,"thread_start_us":722,"threads_started":1}
I20260924 04:08:23.352792 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:23.345402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46317 (local address 127.0.0.1:36668)
0924 04:08:23.347455 (+ 2053us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:23.347468 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:23.347557 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:23.347978 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:23.347982 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:23.347998 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:08:23.348103 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:23.348109 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:23.349299 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:23.349302 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:23.350393 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:23.350402 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:23.352363 (+ 1961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:23.352367 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:23.352456 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:23.352478 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 04:08:23.352548 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 04:08:23.352617 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1992,"mutex_wait_us":1,"thread_start_us":100,"threads_started":1}
I20260924 04:08:23.353350 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:23.345820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36668 (local address 127.0.98.3:46317)
0924 04:08:23.346702 (+ 882us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:23.346711 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:23.347566 (+ 855us) server_negotiation.cc:407] Connection header received
0924 04:08:23.347652 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:23.347660 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:23.347838 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:23.347987 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:08:23.348198 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:23.349185 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:23.350583 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:23.352063 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:23.352161 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:23.352298 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:23.353112 (+ 814us) server_negotiation.cc:299] Negotiation successful
0924 04:08:23.353204 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":782,"thread_start_us":264,"threads_started":1}
I20260924 04:08:23.354055 1221 tablet_service.cc:1467] Processing CreateTablet for tablet b3646ddde19f4dc1ac42b9b5d42e512f (DEFAULT_TABLE table=test-table [id=1c22b535be344a079f781086fb37731d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:08:23.354405 1087 tablet_service.cc:1467] Processing CreateTablet for tablet b3646ddde19f4dc1ac42b9b5d42e512f (DEFAULT_TABLE table=test-table [id=1c22b535be344a079f781086fb37731d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:08:23.355253 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3646ddde19f4dc1ac42b9b5d42e512f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:08:23.355353 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3646ddde19f4dc1ac42b9b5d42e512f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:08:23.355960 1355 tablet_service.cc:1467] Processing CreateTablet for tablet b3646ddde19f4dc1ac42b9b5d42e512f (DEFAULT_TABLE table=test-table [id=1c22b535be344a079f781086fb37731d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:08:23.356926 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3646ddde19f4dc1ac42b9b5d42e512f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:08:23.361991 1438 tablet_bootstrap.cc:492] T b3646ddde19f4dc1ac42b9b5d42e512f P 277c3fe31eca45a983c87b1df46dd0f7: Bootstrap starting.
I20260924 04:08:23.363329 1439 tablet_bootstrap.cc:492] T b3646ddde19f4dc1ac42b9b5d42e512f P f2c8bf16eb554b0daf7a1854b87cffcb: Bootstrap starting.
I20260924 04:08:23.364451 1438 tablet_bootstrap.cc:654] T b3646ddde19f4dc1ac42b9b5d42e512f P 277c3fe31eca45a983c87b1df46dd0f7: Neither blocks nor log segments found. Creating new log.
I20260924 04:08:23.365918 1439 tablet_bootstrap.cc:654] T b3646ddde19f4dc1ac42b9b5d42e512f P f2c8bf16eb554b0daf7a1854b87cffcb: Neither blocks nor log segments found. Creating new log.
I20260924 04:08:23.365888 1438 log.cc:824] T b3646ddde19f4dc1ac42b9b5d42e512f P 277c3fe31eca45a983c87b1df46dd0f7: Log is configured to *not* fsync() on all Append() calls
I20260924 04:08:23.366719 1439 log.cc:824] T b3646ddde19f4dc1ac42b9b5d42e512f P f2c8bf16eb554b0daf7a1854b87cffcb: Log is configured to *not* fsync() on all Append() calls
I20260924 04:08:23.367080 1440 tablet_bootstrap.cc:492] T b3646ddde19f4dc1ac42b9b5d42e512f P 02113ec861ca4024a5bf8c09c2728c9f: Bootstrap starting.
I20260924 04:08:23.368045 1438 tablet_bootstrap.cc:492] T b3646ddde19f4dc1ac42b9b5d42e512f P 277c3fe31eca45a983c87b1df46dd0f7: No bootstrap required, opened a new log
I20260924 04:08:23.368261 1438 ts_tablet_manager.cc:1397] T b3646ddde19f4dc1ac42b9b5d42e512f P 277c3fe31eca45a983c87b1df46dd0f7: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 04:08:23.368835 1439 tablet_bootstrap.cc:492] T b3646ddde19f4dc1ac42b9b5d42e512f P f2c8bf16eb554b0daf7a1854b87cffcb: No bootstrap required, opened a new log
I20260924 04:08:23.369019 1439 ts_tablet_manager.cc:1397] T b3646ddde19f4dc1ac42b9b5d42e512f P f2c8bf16eb554b0daf7a1854b87cffcb: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 04:08:23.369134 1440 tablet_bootstrap.cc:654] T b3646ddde19f4dc1ac42b9b5d42e512f P 02113ec861ca4024a5bf8c09c2728c9f: Neither blocks nor log segments found. Creating new log.
I20260924 04:08:23.370324 1440 log.cc:824] T b3646ddde19f4dc1ac42b9b5d42e512f P 02113ec861ca4024a5bf8c09c2728c9f: Log is configured to *not* fsync() on all Append() calls
I20260924 04:08:23.371685 1438 raft_consensus.cc:348] T b3646ddde19f4dc1ac42b9b5d42e512f P 277c3fe31eca45a983c87b1df46dd0f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2c8bf16eb554b0daf7a1854b87cffcb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38305 } } peers { permanent_uuid: "02113ec861ca4024a5bf8c09c2728c9f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46317 } } peers { permanent_uuid: "277c3fe31eca45a983c87b1df46dd0f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43681 } }
I20260924 04:08:23.371925 1438 raft_consensus.cc:374] T b3646ddde19f4dc1ac42b9b5d42e512f P 277c3fe31eca45a983c87b1df46dd0f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:08:23.371979 1438 raft_consensus.cc:729] T b3646ddde19f4dc1ac42b9b5d42e512f P 277c3fe31eca45a983c87b1df46dd0f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 277c3fe31eca45a983c87b1df46dd0f7, State: Initialized, Role: FOLLOWER
I20260924 04:08:23.372442 1438 consensus_queue.cc:260] T b3646ddde19f4dc1ac42b9b5d42e512f P 277c3fe31eca45a983c87b1df46dd0f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2c8bf16eb554b0daf7a1854b87cffcb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38305 } } peers { permanent_uuid: "02113ec861ca4024a5bf8c09c2728c9f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46317 } } peers { permanent_uuid: "277c3fe31eca45a983c87b1df46dd0f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43681 } }
I20260924 04:08:23.372603 1440 tablet_bootstrap.cc:492] T b3646ddde19f4dc1ac42b9b5d42e512f P 02113ec861ca4024a5bf8c09c2728c9f: No bootstrap required, opened a new log
I20260924 04:08:23.372808 1440 ts_tablet_manager.cc:1397] T b3646ddde19f4dc1ac42b9b5d42e512f P 02113ec861ca4024a5bf8c09c2728c9f: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 04:08:23.373188 1439 raft_consensus.cc:348] T b3646ddde19f4dc1ac42b9b5d42e512f P f2c8bf16eb554b0daf7a1854b87cffcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2c8bf16eb554b0daf7a1854b87cffcb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38305 } } peers { permanent_uuid: "02113ec861ca4024a5bf8c09c2728c9f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46317 } } peers { permanent_uuid: "277c3fe31eca45a983c87b1df46dd0f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43681 } }
I20260924 04:08:23.373456 1439 raft_consensus.cc:374] T b3646ddde19f4dc1ac42b9b5d42e512f P f2c8bf16eb554b0daf7a1854b87cffcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:08:23.373476 1438 ts_tablet_manager.cc:1428] T b3646ddde19f4dc1ac42b9b5d42e512f P 277c3fe31eca45a983c87b1df46dd0f7: Time spent starting tablet: real 0.005s user 0.003s sys 0.004s
I20260924 04:08:23.373504 1439 raft_consensus.cc:729] T b3646ddde19f4dc1ac42b9b5d42e512f P f2c8bf16eb554b0daf7a1854b87cffcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2c8bf16eb554b0daf7a1854b87cffcb, State: Initialized, Role: FOLLOWER
I20260924 04:08:23.373811 1153 heartbeater.cc:503] Master 127.0.98.62:41957 was elected leader, sending a full tablet report...
I20260924 04:08:23.373939 1439 consensus_queue.cc:260] T b3646ddde19f4dc1ac42b9b5d42e512f P f2c8bf16eb554b0daf7a1854b87cffcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2c8bf16eb554b0daf7a1854b87cffcb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38305 } } peers { permanent_uuid: "02113ec861ca4024a5bf8c09c2728c9f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46317 } } peers { permanent_uuid: "277c3fe31eca45a983c87b1df46dd0f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43681 } }
I20260924 04:08:23.374568 1287 heartbeater.cc:503] Master 127.0.98.62:41957 was elected leader, sending a full tablet report...
I20260924 04:08:23.376111 1439 ts_tablet_manager.cc:1428] T b3646ddde19f4dc1ac42b9b5d42e512f P f2c8bf16eb554b0daf7a1854b87cffcb: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260924 04:08:23.376418 1440 raft_consensus.cc:348] T b3646ddde19f4dc1ac42b9b5d42e512f P 02113ec861ca4024a5bf8c09c2728c9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2c8bf16eb554b0daf7a1854b87cffcb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38305 } } peers { permanent_uuid: "02113ec861ca4024a5bf8c09c2728c9f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46317 } } peers { permanent_uuid: "277c3fe31eca45a983c87b1df46dd0f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43681 } }
I20260924 04:08:23.376675 1440 raft_consensus.cc:374] T b3646ddde19f4dc1ac42b9b5d42e512f P 02113ec861ca4024a5bf8c09c2728c9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:08:23.376950 1440 raft_consensus.cc:729] T b3646ddde19f4dc1ac42b9b5d42e512f P 02113ec861ca4024a5bf8c09c2728c9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02113ec861ca4024a5bf8c09c2728c9f, State: Initialized, Role: FOLLOWER
I20260924 04:08:23.377405 1440 consensus_queue.cc:260] T b3646ddde19f4dc1ac42b9b5d42e512f P 02113ec861ca4024a5bf8c09c2728c9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2c8bf16eb554b0daf7a1854b87cffcb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38305 } } peers { permanent_uuid: "02113ec861ca4024a5bf8c09c2728c9f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46317 } } peers { permanent_uuid: "277c3fe31eca45a983c87b1df46dd0f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43681 } }
I20260924 04:08:23.378334 1440 ts_tablet_manager.cc:1428] T b3646ddde19f4dc1ac42b9b5d42e512f P 02113ec861ca4024a5bf8c09c2728c9f: Time spent starting tablet: real 0.005s user 0.001s sys 0.003s
I20260924 04:08:23.378676 1421 heartbeater.cc:503] Master 127.0.98.62:41957 was elected leader, sending a full tablet report...
I20260924 04:08:23.528106 1444 raft_consensus.cc:482] T b3646ddde19f4dc1ac42b9b5d42e512f P 277c3fe31eca45a983c87b1df46dd0f7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:08:23.528307 1444 raft_consensus.cc:504] T b3646ddde19f4dc1ac42b9b5d42e512f P 277c3fe31eca45a983c87b1df46dd0f7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2c8bf16eb554b0daf7a1854b87cffcb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38305 } } peers { permanent_uuid: "02113ec861ca4024a5bf8c09c2728c9f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46317 } } peers { permanent_uuid: "277c3fe31eca45a983c87b1df46dd0f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43681 } }
I20260924 04:08:23.529449 1444 leader_election.cc:288] T b3646ddde19f4dc1ac42b9b5d42e512f P 277c3fe31eca45a983c87b1df46dd0f7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f2c8bf16eb554b0daf7a1854b87cffcb (127.0.98.2:38305), 02113ec861ca4024a5bf8c09c2728c9f (127.0.98.3:46317)
I20260924 04:08:23.533592 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:23.529881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35587 (local address 127.0.98.2:38305)
0924 04:08:23.530015 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:23.530021 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:23.530101 (+ 80us) server_negotiation.cc:407] Connection header received
0924 04:08:23.530283 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:23.530289 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:23.530346 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:23.530422 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:08:23.530842 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:23.531523 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:23.532574 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:23.533054 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:23.533179 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:23.533243 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:23.533408 (+ 165us) server_negotiation.cc:299] Negotiation successful
0924 04:08:23.533470 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 04:08:23.533604 1447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:23.529737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38305 (local address 127.0.98.1:35587)
0924 04:08:23.530014 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:23.530031 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:23.530158 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:23.530565 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:23.530568 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:23.530595 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:08:23.530745 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:23.530753 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:23.531638 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:23.531644 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:23.532448 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:23.532457 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:23.533177 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:23.533180 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:23.533277 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:23.533299 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 04:08:23.533394 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 04:08:23.533463 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":123,"threads_started":1}
I20260924 04:08:23.533728 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:23.530473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59455 (local address 127.0.98.3:46317)
0924 04:08:23.530756 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:23.530760 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:23.530774 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:08:23.530825 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:23.530827 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:23.530872 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:23.530948 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:08:23.531232 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:23.531686 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:23.532763 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:23.533178 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:23.533293 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:23.533351 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:23.533534 (+ 183us) server_negotiation.cc:299] Negotiation successful
0924 04:08:23.533595 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155}
I20260924 04:08:23.533775 1448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:23.530166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46317 (local address 127.0.98.1:59455)
0924 04:08:23.530568 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:23.530580 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:23.530671 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:23.530978 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:23.530981 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:23.531000 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:08:23.531130 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:23.531136 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:23.531846 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:23.531849 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:23.532610 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:23.532618 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:23.533298 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:23.533301 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:23.533413 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:23.533430 (+ 17us) client_negotiation.cc:769] Sending connection context
0924 04:08:23.533517 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 04:08:23.533594 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":95,"threads_started":1}
I20260924 04:08:23.534399 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3646ddde19f4dc1ac42b9b5d42e512f" candidate_uuid: "277c3fe31eca45a983c87b1df46dd0f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02113ec861ca4024a5bf8c09c2728c9f" is_pre_election: true
I20260924 04:08:23.534636 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3646ddde19f4dc1ac42b9b5d42e512f" candidate_uuid: "277c3fe31eca45a983c87b1df46dd0f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2c8bf16eb554b0daf7a1854b87cffcb" is_pre_election: true
I20260924 04:08:23.534723 1375 raft_consensus.cc:2454] T b3646ddde19f4dc1ac42b9b5d42e512f P 02113ec861ca4024a5bf8c09c2728c9f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 277c3fe31eca45a983c87b1df46dd0f7 in term 0.
I20260924 04:08:23.534888 1241 raft_consensus.cc:2454] T b3646ddde19f4dc1ac42b9b5d42e512f P f2c8bf16eb554b0daf7a1854b87cffcb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 277c3fe31eca45a983c87b1df46dd0f7 in term 0.
I20260924 04:08:23.535212 1040 leader_election.cc:302] T b3646ddde19f4dc1ac42b9b5d42e512f P 277c3fe31eca45a983c87b1df46dd0f7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02113ec861ca4024a5bf8c09c2728c9f, 277c3fe31eca45a983c87b1df46dd0f7; no voters:
I20260924 04:08:23.535434 1444 raft_consensus.cc:2781] T b3646ddde19f4dc1ac42b9b5d42e512f P 277c3fe31eca45a983c87b1df46dd0f7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:08:23.535529 1444 raft_consensus.cc:482] T b3646ddde19f4dc1ac42b9b5d42e512f P 277c3fe31eca45a983c87b1df46dd0f7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:08:23.535586 1444 raft_consensus.cc:3037] T b3646ddde19f4dc1ac42b9b5d42e512f P 277c3fe31eca45a983c87b1df46dd0f7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:08:23.536588 1444 raft_consensus.cc:504] T b3646ddde19f4dc1ac42b9b5d42e512f P 277c3fe31eca45a983c87b1df46dd0f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2c8bf16eb554b0daf7a1854b87cffcb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38305 } } peers { permanent_uuid: "02113ec861ca4024a5bf8c09c2728c9f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46317 } } peers { permanent_uuid: "277c3fe31eca45a983c87b1df46dd0f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43681 } }
I20260924 04:08:23.536939 1444 leader_election.cc:288] T b3646ddde19f4dc1ac42b9b5d42e512f P 277c3fe31eca45a983c87b1df46dd0f7 [CANDIDATE]: Term 1 election: Requested vote from peers f2c8bf16eb554b0daf7a1854b87cffcb (127.0.98.2:38305), 02113ec861ca4024a5bf8c09c2728c9f (127.0.98.3:46317)
I20260924 04:08:23.537276 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3646ddde19f4dc1ac42b9b5d42e512f" candidate_uuid: "277c3fe31eca45a983c87b1df46dd0f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2c8bf16eb554b0daf7a1854b87cffcb"
I20260924 04:08:23.537354 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3646ddde19f4dc1ac42b9b5d42e512f" candidate_uuid: "277c3fe31eca45a983c87b1df46dd0f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02113ec861ca4024a5bf8c09c2728c9f"
I20260924 04:08:23.537398 1241 raft_consensus.cc:3037] T b3646ddde19f4dc1ac42b9b5d42e512f P f2c8bf16eb554b0daf7a1854b87cffcb [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:08:23.537488 1375 raft_consensus.cc:3037] T b3646ddde19f4dc1ac42b9b5d42e512f P 02113ec861ca4024a5bf8c09c2728c9f [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:08:23.538439 1375 raft_consensus.cc:2454] T b3646ddde19f4dc1ac42b9b5d42e512f P 02113ec861ca4024a5bf8c09c2728c9f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 277c3fe31eca45a983c87b1df46dd0f7 in term 1.
I20260924 04:08:23.538439 1241 raft_consensus.cc:2454] T b3646ddde19f4dc1ac42b9b5d42e512f P f2c8bf16eb554b0daf7a1854b87cffcb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 277c3fe31eca45a983c87b1df46dd0f7 in term 1.
I20260924 04:08:23.538733 1040 leader_election.cc:302] T b3646ddde19f4dc1ac42b9b5d42e512f P 277c3fe31eca45a983c87b1df46dd0f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02113ec861ca4024a5bf8c09c2728c9f, 277c3fe31eca45a983c87b1df46dd0f7; no voters:
I20260924 04:08:23.538898 1444 raft_consensus.cc:2781] T b3646ddde19f4dc1ac42b9b5d42e512f P 277c3fe31eca45a983c87b1df46dd0f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:08:23.539114 1444 raft_consensus.cc:686] T b3646ddde19f4dc1ac42b9b5d42e512f P 277c3fe31eca45a983c87b1df46dd0f7 [term 1 LEADER]: Becoming Leader. State: Replica: 277c3fe31eca45a983c87b1df46dd0f7, State: Running, Role: LEADER
I20260924 04:08:23.539434 1444 consensus_queue.cc:237] T b3646ddde19f4dc1ac42b9b5d42e512f P 277c3fe31eca45a983c87b1df46dd0f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2c8bf16eb554b0daf7a1854b87cffcb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38305 } } peers { permanent_uuid: "02113ec861ca4024a5bf8c09c2728c9f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46317 } } peers { permanent_uuid: "277c3fe31eca45a983c87b1df46dd0f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43681 } }
I20260924 04:08:23.542450 961 catalog_manager.cc:5495] T b3646ddde19f4dc1ac42b9b5d42e512f P 277c3fe31eca45a983c87b1df46dd0f7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 277c3fe31eca45a983c87b1df46dd0f7 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "277c3fe31eca45a983c87b1df46dd0f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2c8bf16eb554b0daf7a1854b87cffcb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02113ec861ca4024a5bf8c09c2728c9f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46317 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "277c3fe31eca45a983c87b1df46dd0f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43681 } health_report { overall_health: HEALTHY } } }
W20260924 04:08:23.544911 1422 tablet.cc:2369] T b3646ddde19f4dc1ac42b9b5d42e512f P 02113ec861ca4024a5bf8c09c2728c9f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 04:08:23.578497 1288 tablet.cc:2369] T b3646ddde19f4dc1ac42b9b5d42e512f P f2c8bf16eb554b0daf7a1854b87cffcb: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 04:08:23 dist-test-slave-00ld krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222902, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 04:08:23.593222 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:23.578266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37210 (local address 127.0.98.1:43681)
0924 04:08:23.578401 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:23.578406 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:23.584232 (+ 5826us) server_negotiation.cc:407] Connection header received
0924 04:08:23.584351 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:23.584354 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:23.584425 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:23.584496 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:23.585322 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:23.585836 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:23.586815 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:23.587066 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:23.587139 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:23.589704 (+ 2565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:23.589717 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:23.589731 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:23.589753 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:23.591501 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:23.591945 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:23.591950 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:23.591952 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:23.592031 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:23.592312 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:23.592315 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:23.592317 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:23.592549 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:23.592797 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:23.592943 (+ 146us) server_negotiation.cc:299] Negotiation successful
0924 04:08:23.593059 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
Sep 24 04:08:23 dist-test-slave-00ld krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222903, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 04:08:23 dist-test-slave-00ld krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222903, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
W20260924 04:08:23.618374 1154 tablet.cc:2369] T b3646ddde19f4dc1ac42b9b5d42e512f P 277c3fe31eca45a983c87b1df46dd0f7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:08:23.620064 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:23.611240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37214 (local address 127.0.98.1:43681)
0924 04:08:23.611420 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:23.611425 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:23.611563 (+ 138us) server_negotiation.cc:407] Connection header received
0924 04:08:23.611759 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:23.611762 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:23.611809 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:23.611907 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:23.612566 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:23.613240 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:23.614065 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:23.614304 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:23.614410 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:23.616902 (+ 2492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:23.616905 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:23.616907 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:23.616929 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:23.618556 (+ 1627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:23.618962 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:23.618964 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:23.618965 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:23.619034 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:23.619278 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:23.619280 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:23.619281 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:23.619424 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:23.619517 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:23.619764 (+ 247us) server_negotiation.cc:299] Negotiation successful
0924 04:08:23.619881 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
Sep 24 04:08:23 dist-test-slave-00ld krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222903, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 04:08:23 dist-test-slave-00ld krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222903, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 04:08:23.644984 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:23.636891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37228 (local address 127.0.98.1:43681)
0924 04:08:23.637040 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:23.637047 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:23.637179 (+ 132us) server_negotiation.cc:407] Connection header received
0924 04:08:23.637294 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:23.637297 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:23.637340 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:23.637411 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:23.638163 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:23.638659 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:23.639447 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:23.639647 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:23.639725 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:23.642177 (+ 2452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:23.642180 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:23.642182 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:23.642206 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:23.643470 (+ 1264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:23.643896 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:23.643898 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:23.643900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:23.643973 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:23.644202 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:23.644204 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:23.644206 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:23.644358 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:23.644469 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:23.644768 (+ 299us) server_negotiation.cc:299] Negotiation successful
0924 04:08:23.644865 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260924 04:08:23.646306 1067 tablet_service.cc:2908] Rejecting scan request for tablet b3646ddde19f4dc1ac42b9b5d42e512f: Uninitialized: clean time has not yet been initialized
I20260924 04:08:23.648792 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 1024
I20260924 04:08:23.655117 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 1158
I20260924 04:08:23.660437 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 1292
I20260924 04:08:23.665925 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 930
2026-09-24T04:08:23Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2660 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222898807720-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:08:23 dist-test-slave-00ld krb5kdc[1463](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:08:23 dist-test-slave-00ld krb5kdc[1463](info): set up 2 sockets
Sep 24 04:08:23 dist-test-slave-00ld krb5kdc[1463](info): commencing operation
krb5kdc: starting...
W20260924 04:08:24.705036 392 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222898807720-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222898807720-392-0/krb5kdc/test-user.keytab.
Sep 24 04:08:24 dist-test-slave-00ld krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222904, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:08:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:08:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:08:24.831903 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222898807720-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222898807720-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222898807720-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40335
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40335
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:08:24.943519 1476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:08:24.943889 1476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:08:24.943970 1476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:08:24.948455 1476 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:08:24.948594 1476 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 04:08:24.948632 1476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:08:24.948704 1476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:08:24.948766 1476 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:08:24.948807 1476 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:08:24.954553 1476 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222898807720-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222898807720-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40335
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40335
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222898807720-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:08:24.955717 1476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:08:24.956944 1476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:08:24.965478 1481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:08:24.965503 1484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:08:24.965564 1482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:08:24.965559 1476 server_base.cc:1034] running on GCE node
I20260924 04:08:24.966266 1476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:08:24.967398 1476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:08:24.968581 1476 hybrid_clock.cc:648] HybridClock initialized: now 1790222904968548 us; error 44 us; skew 500 ppm
Sep 24 04:08:24 dist-test-slave-00ld krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222904, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:08:24.975780 1476 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:08:24.976830 1476 webserver.cc:460] Webserver started at http://127.0.98.62:39283/ using document root <none> and password file <none>
I20260924 04:08:24.977456 1476 fs_manager.cc:359] Metadata directory not provided
I20260924 04:08:24.977555 1476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:08:24.977802 1476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:08:24.980033 1476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222898807720-392-0/minicluster-data/master-0/data/instance:
uuid: "a9550222f0fa4b488486b4d4ff95b688"
format_stamp: "Formatted at 2026-09-24 04:08:24 on dist-test-slave-00ld"
server_key: "7d1247573b2976682350eab5e71fa475"
server_key_iv: "dc6b21e76e8e540f416f4aa42164a3a1"
server_key_version: "encryptionkey@0"
I20260924 04:08:24.980744 1476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222898807720-392-0/minicluster-data/master-0/wal/instance:
uuid: "a9550222f0fa4b488486b4d4ff95b688"
format_stamp: "Formatted at 2026-09-24 04:08:24 on dist-test-slave-00ld"
server_key: "7d1247573b2976682350eab5e71fa475"
server_key_iv: "dc6b21e76e8e540f416f4aa42164a3a1"
server_key_version: "encryptionkey@0"
I20260924 04:08:24.984844 1476 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260924 04:08:24.987427 1491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:24.988974 1476 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:08:24.989133 1476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222898807720-392-0/minicluster-data/master-0/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222898807720-392-0/minicluster-data/master-0/wal
uuid: "a9550222f0fa4b488486b4d4ff95b688"
format_stamp: "Formatted at 2026-09-24 04:08:24 on dist-test-slave-00ld"
server_key: "7d1247573b2976682350eab5e71fa475"
server_key_iv: "dc6b21e76e8e540f416f4aa42164a3a1"
server_key_version: "encryptionkey@0"
I20260924 04:08:24.989281 1476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222898807720-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222898807720-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222898807720-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:08:25.007148 1476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:08:25.010648 1476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:08:25.010876 1476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:08:25.019928 1476 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40335
I20260924 04:08:25.019945 1553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40335 every 8 connection(s)
I20260924 04:08:25.021024 1476 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:08:25.023887 1554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:08:25.028962 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 1476
I20260924 04:08:25.029060 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222898807720-392-0/minicluster-data/master-0/wal/instance
I20260924 04:08:25.029325 392 external_mini_cluster.cc:1428] Setting key 57386d7d11035c42097ac09fcd358e5f
I20260924 04:08:25.029670 1554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9550222f0fa4b488486b4d4ff95b688: Bootstrap starting.
I20260924 04:08:25.032250 1554 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a9550222f0fa4b488486b4d4ff95b688: Neither blocks nor log segments found. Creating new log.
I20260924 04:08:25.033069 1554 log.cc:824] T 00000000000000000000000000000000 P a9550222f0fa4b488486b4d4ff95b688: Log is configured to *not* fsync() on all Append() calls
I20260924 04:08:25.034932 1554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9550222f0fa4b488486b4d4ff95b688: No bootstrap required, opened a new log
Sep 24 04:08:25 dist-test-slave-00ld krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222904, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:08:25.038936 1554 raft_consensus.cc:348] T 00000000000000000000000000000000 P a9550222f0fa4b488486b4d4ff95b688 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9550222f0fa4b488486b4d4ff95b688" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40335 } }
I20260924 04:08:25.039183 1554 raft_consensus.cc:374] T 00000000000000000000000000000000 P a9550222f0fa4b488486b4d4ff95b688 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:08:25.039247 1554 raft_consensus.cc:729] T 00000000000000000000000000000000 P a9550222f0fa4b488486b4d4ff95b688 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9550222f0fa4b488486b4d4ff95b688, State: Initialized, Role: FOLLOWER
I20260924 04:08:25.039743 1554 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9550222f0fa4b488486b4d4ff95b688 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9550222f0fa4b488486b4d4ff95b688" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40335 } }
I20260924 04:08:25.039911 1554 raft_consensus.cc:388] T 00000000000000000000000000000000 P a9550222f0fa4b488486b4d4ff95b688 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:08:25.039987 1554 raft_consensus.cc:482] T 00000000000000000000000000000000 P a9550222f0fa4b488486b4d4ff95b688 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:08:25.040102 1554 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a9550222f0fa4b488486b4d4ff95b688 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:08:25.041090 1554 raft_consensus.cc:504] T 00000000000000000000000000000000 P a9550222f0fa4b488486b4d4ff95b688 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9550222f0fa4b488486b4d4ff95b688" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40335 } }
I20260924 04:08:25.041442 1554 leader_election.cc:302] T 00000000000000000000000000000000 P a9550222f0fa4b488486b4d4ff95b688 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a9550222f0fa4b488486b4d4ff95b688; no voters:
I20260924 04:08:25.041761 1554 leader_election.cc:288] T 00000000000000000000000000000000 P a9550222f0fa4b488486b4d4ff95b688 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:08:25.042119 1559 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a9550222f0fa4b488486b4d4ff95b688 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:08:25.042336 1559 raft_consensus.cc:686] T 00000000000000000000000000000000 P a9550222f0fa4b488486b4d4ff95b688 [term 1 LEADER]: Becoming Leader. State: Replica: a9550222f0fa4b488486b4d4ff95b688, State: Running, Role: LEADER
I20260924 04:08:25.042654 1559 consensus_queue.cc:237] T 00000000000000000000000000000000 P a9550222f0fa4b488486b4d4ff95b688 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9550222f0fa4b488486b4d4ff95b688" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40335 } }
I20260924 04:08:25.043159 1554 sys_catalog.cc:564] T 00000000000000000000000000000000 P a9550222f0fa4b488486b4d4ff95b688 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:08:25.044305 1560 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9550222f0fa4b488486b4d4ff95b688 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a9550222f0fa4b488486b4d4ff95b688" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9550222f0fa4b488486b4d4ff95b688" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40335 } } }
I20260924 04:08:25.044422 1560 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9550222f0fa4b488486b4d4ff95b688 [sys.catalog]: This master's current role is: LEADER
I20260924 04:08:25.044670 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:25.030731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34076 (local address 127.0.98.62:40335)
0924 04:08:25.031220 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:25.031235 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:25.031270 (+ 35us) server_negotiation.cc:407] Connection header received
0924 04:08:25.031953 (+ 683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:25.031970 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:25.032320 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:25.032765 (+ 445us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:25.033618 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:25.034598 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:25.035490 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:25.035976 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:25.036166 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:25.039006 (+ 2840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:25.039012 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:25.039025 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:25.039050 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:25.040978 (+ 1928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:25.041524 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:25.041529 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:25.041533 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:25.041695 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:25.041989 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:25.041995 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:25.041997 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:25.042615 (+ 618us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:25.042815 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:25.043108 (+ 293us) server_negotiation.cc:299] Negotiation successful
0924 04:08:25.043596 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":152,"threads_started":1}
I20260924 04:08:25.044760 1561 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9550222f0fa4b488486b4d4ff95b688 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a9550222f0fa4b488486b4d4ff95b688. Latest consensus state: current_term: 1 leader_uuid: "a9550222f0fa4b488486b4d4ff95b688" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9550222f0fa4b488486b4d4ff95b688" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40335 } } }
I20260924 04:08:25.044829 1561 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9550222f0fa4b488486b4d4ff95b688 [sys.catalog]: This master's current role is: LEADER
I20260924 04:08:25.045076 1568 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:08:25.047885 1568 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:08:25.053900 1568 catalog_manager.cc:1347] Generated new cluster ID: 463f4033bf1d426f87ebe795b7d7d991
I20260924 04:08:25.053985 1568 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:08:25.065717 1568 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:08:25.067036 1568 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:08:25.072443 1568 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a9550222f0fa4b488486b4d4ff95b688: Generated new TSK 0
I20260924 04:08:25.073276 1568 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222898807720-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222898807720-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:08:25.108322 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222898807720-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222898807720-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222898807720-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222898807720-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40335
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222898807720-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:08:25.221577 1580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:08:25.221838 1580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:08:25.221910 1580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:08:25.226091 1580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:08:25.226150 1580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:08:25.226188 1580 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 04:08:25.226271 1580 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:08:25.231412 1580 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222898807720-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222898807720-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222898807720-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222898807720-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:40335
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1580
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222898807720-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:08:25.232487 1580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:08:25.233520 1580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:08:25.243116 1580 server_base.cc:1034] running on GCE node
W20260924 04:08:25.243052 1585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:08:25.243027 1588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:08:25.243052 1586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:08:25.243721 1580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:08:25.244433 1580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:08:25.245646 1580 hybrid_clock.cc:648] HybridClock initialized: now 1790222905245633 us; error 39 us; skew 500 ppm
Sep 24 04:08:25 dist-test-slave-00ld krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222905, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:08:25.248564 1580 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:08:25.249581 1580 webserver.cc:460] Webserver started at http://127.0.98.1:44293/ using document root <none> and password file <none>
I20260924 04:08:25.250149 1580 fs_manager.cc:359] Metadata directory not provided
I20260924 04:08:25.250239 1580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:08:25.250473 1580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:08:25.252490 1580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222898807720-392-0/minicluster-data/ts-0/data/instance:
uuid: "7db0dad5cd924a5f80a51795d2a0315d"
format_stamp: "Formatted at 2026-09-24 04:08:25 on dist-test-slave-00ld"
server_key: "ca0093f31afb4836d60bb6d70fcd4652"
server_key_iv: "7f1b3d8dd9f86feac70969a0ebea0c44"
server_key_version: "encryptionkey@0"
I20260924 04:08:25.253098 1580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222898807720-392-0/minicluster-data/ts-0/wal/instance:
uuid: "7db0dad5cd924a5f80a51795d2a0315d"
format_stamp: "Formatted at 2026-09-24 04:08:25 on dist-test-slave-00ld"
server_key: "ca0093f31afb4836d60bb6d70fcd4652"
server_key_iv: "7f1b3d8dd9f86feac70969a0ebea0c44"
server_key_version: "encryptionkey@0"
I20260924 04:08:25.256836 1580 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:08:25.259449 1595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:25.260669 1580 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:08:25.260869 1580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222898807720-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222898807720-392-0/minicluster-data/ts-0/wal
uuid: "7db0dad5cd924a5f80a51795d2a0315d"
format_stamp: "Formatted at 2026-09-24 04:08:25 on dist-test-slave-00ld"
server_key: "ca0093f31afb4836d60bb6d70fcd4652"
server_key_iv: "7f1b3d8dd9f86feac70969a0ebea0c44"
server_key_version: "encryptionkey@0"
I20260924 04:08:25.260982 1580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222898807720-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222898807720-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222898807720-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:08:25.270167 1580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:08:25.272996 1580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:08:25.273191 1580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:08:25.274681 1580 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:08:25.274752 1580 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:25.274824 1580 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:08:25.274869 1580 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 04:08:25 dist-test-slave-00ld krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222905, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:08:25.285792 1580 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41777
I20260924 04:08:25.286888 1580 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222898807720-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:08:25.287839 1710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41777 every 8 connection(s)
I20260924 04:08:25.288884 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:25.275520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55137 (local address 127.0.98.62:40335)
0924 04:08:25.275692 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:25.275695 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:25.276674 (+ 979us) server_negotiation.cc:407] Connection header received
0924 04:08:25.277676 (+ 1002us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:25.277684 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:25.277754 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:25.277850 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:25.279156 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:25.279792 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:25.281077 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:25.281311 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:25.281423 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:25.284894 (+ 3471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:25.284899 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:25.284902 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:25.284948 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:25.286625 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:25.287218 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:25.287222 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:25.287225 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:25.287320 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:25.287751 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:25.287754 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:25.287756 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:25.288000 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:25.288116 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:25.288595 (+ 479us) server_negotiation.cc:299] Negotiation successful
0924 04:08:25.288749 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260924 04:08:25.289551 1604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:25.276056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40335 (local address 127.0.98.1:55137)
0924 04:08:25.276526 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:25.276574 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:25.277413 (+ 839us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:25.278100 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:25.278110 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:25.278443 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:08:25.279011 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:25.279025 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:25.279919 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:25.279922 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:25.280954 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:25.280962 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:25.281457 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:25.281461 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:25.281629 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:25.282268 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:08:25.282293 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:08:25.284669 (+ 2376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:08:25.286751 (+ 2082us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:25.286756 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:25.286771 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:25.287099 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:25.287440 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:25.287444 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:25.287446 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:25.287622 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:25.288160 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:08:25.288165 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:08:25.288367 (+ 202us) client_negotiation.cc:769] Sending connection context
0924 04:08:25.288509 (+ 142us) client_negotiation.cc:240] Negotiation successful
0924 04:08:25.288803 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":134,"threads_started":1}
I20260924 04:08:25.290789 1711 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40335
I20260924 04:08:25.290942 1711 heartbeater.cc:464] Registering TS with master...
I20260924 04:08:25.291419 1711 heartbeater.cc:511] Master 127.0.98.62:40335 requested a full tablet report, sending...
I20260924 04:08:25.292964 1504 ts_manager.cc:194] Registered new tserver with Master: 7db0dad5cd924a5f80a51795d2a0315d (127.0.98.1:41777)
I20260924 04:08:25.294377 1504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:55137
I20260924 04:08:25.294821 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 1580
I20260924 04:08:25.294934 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222898807720-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:08:25.295218 392 external_mini_cluster.cc:1428] Setting key e02ab9d930d1621cfc219cfd25e76c78
I20260924 04:08:25.300141 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:25.292809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46001 (local address 127.0.98.62:40335)
0924 04:08:25.292935 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:25.292938 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:25.293178 (+ 240us) server_negotiation.cc:407] Connection header received
0924 04:08:25.293307 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:25.293310 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:25.293352 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:25.293423 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:25.293879 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:25.294538 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:25.295675 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:25.295981 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:25.296093 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:25.297261 (+ 1168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:25.297264 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:25.297267 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:25.297289 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:25.298600 (+ 1311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:25.299001 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:25.299006 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:25.299008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:25.299097 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:25.299371 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:25.299374 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:25.299375 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:25.299535 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:25.299637 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:25.299888 (+ 251us) server_negotiation.cc:299] Negotiation successful
0924 04:08:25.299987 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":53}
I20260924 04:08:25.300149 1717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:25.292746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40335 (local address 127.0.98.1:46001)
0924 04:08:25.293058 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:25.293082 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:25.293202 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:25.293463 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:25.293466 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:25.293664 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:08:25.293775 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:25.293782 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:25.294726 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:25.294729 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:25.295532 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:25.295541 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:25.296108 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:25.296115 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:25.296207 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:25.296730 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:08:25.296745 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:08:25.297135 (+ 390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:08:25.298709 (+ 1574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:25.298714 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:25.298716 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:25.298906 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:25.299186 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:25.299190 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:25.299192 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:25.299276 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:25.299628 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:08:25.299631 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:08:25.299725 (+ 94us) client_negotiation.cc:769] Sending connection context
0924 04:08:25.299827 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 04:08:25.299958 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"mutex_wait_us":1,"thread_start_us":110,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222898807720-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222898807720-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:08:25.323079 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222898807720-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222898807720-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222898807720-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222898807720-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40335
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222898807720-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:08:25.435702 1720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:08:25.435953 1720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:08:25.435994 1720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:08:25.440100 1720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:08:25.440161 1720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:08:25.440199 1720 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 04:08:25.440266 1720 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:08:25.445704 1720 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222898807720-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222898807720-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222898807720-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222898807720-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:40335
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1720
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222898807720-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:08:25.446760 1720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:08:25.447785 1720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:08:25.457401 1726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:08:25.457438 1728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:08:25.457617 1725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:08:25.457760 1720 server_base.cc:1034] running on GCE node
I20260924 04:08:25.458222 1720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:08:25.458976 1720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:08:25.460163 1720 hybrid_clock.cc:648] HybridClock initialized: now 1790222905460153 us; error 34 us; skew 500 ppm
Sep 24 04:08:25 dist-test-slave-00ld krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222905, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:08:25.462988 1720 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:08:25.463871 1720 webserver.cc:460] Webserver started at http://127.0.98.2:36183/ using document root <none> and password file <none>
I20260924 04:08:25.464416 1720 fs_manager.cc:359] Metadata directory not provided
I20260924 04:08:25.464504 1720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:08:25.464811 1720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:08:25.466799 1720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222898807720-392-0/minicluster-data/ts-1/data/instance:
uuid: "c1d025e13aef436f94ce8c10ca541edf"
format_stamp: "Formatted at 2026-09-24 04:08:25 on dist-test-slave-00ld"
server_key: "7f700bd8f16f435777521b4a4dd1c227"
server_key_iv: "b15eb8c0082530426c4310ffb563499b"
server_key_version: "encryptionkey@0"
I20260924 04:08:25.467355 1720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222898807720-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c1d025e13aef436f94ce8c10ca541edf"
format_stamp: "Formatted at 2026-09-24 04:08:25 on dist-test-slave-00ld"
server_key: "7f700bd8f16f435777521b4a4dd1c227"
server_key_iv: "b15eb8c0082530426c4310ffb563499b"
server_key_version: "encryptionkey@0"
I20260924 04:08:25.471444 1720 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 04:08:25.473965 1735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:25.475145 1720 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:08:25.475286 1720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222898807720-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222898807720-392-0/minicluster-data/ts-1/wal
uuid: "c1d025e13aef436f94ce8c10ca541edf"
format_stamp: "Formatted at 2026-09-24 04:08:25 on dist-test-slave-00ld"
server_key: "7f700bd8f16f435777521b4a4dd1c227"
server_key_iv: "b15eb8c0082530426c4310ffb563499b"
server_key_version: "encryptionkey@0"
I20260924 04:08:25.475405 1720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222898807720-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222898807720-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222898807720-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:08:25.516592 1720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:08:25.519743 1720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:08:25.519958 1720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:08:25.521456 1720 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:08:25.521528 1720 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:25.521600 1720 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:08:25.521643 1720 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 04:08:25 dist-test-slave-00ld krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222905, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:08:25.532613 1720 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39347
I20260924 04:08:25.533463 1720 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222898807720-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:08:25.533973 1850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39347 every 8 connection(s)
I20260924 04:08:25.535235 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:25.522321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39415 (local address 127.0.98.62:40335)
0924 04:08:25.522447 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:25.522450 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:25.523557 (+ 1107us) server_negotiation.cc:407] Connection header received
0924 04:08:25.524615 (+ 1058us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:25.524618 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:25.524744 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:25.524811 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:25.526303 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:25.527100 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:25.528344 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:25.528639 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:25.528808 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:25.531491 (+ 2683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:25.531496 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:25.531500 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:25.531530 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:25.533118 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:25.533644 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:25.533648 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:25.533650 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:25.533735 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:25.534130 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:25.534134 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:25.534136 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:25.534354 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:25.534451 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:25.535008 (+ 557us) server_negotiation.cc:299] Negotiation successful
0924 04:08:25.535119 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 04:08:25.536048 1744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:25.522890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40335 (local address 127.0.98.2:39415)
0924 04:08:25.523380 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:25.523423 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:25.524364 (+ 941us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:25.525144 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:25.525150 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:25.525549 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:08:25.526141 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:25.526166 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:25.527235 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:25.527238 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:25.528189 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:25.528198 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:25.528860 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:25.528864 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:25.529029 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:25.529638 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:08:25.529659 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:08:25.531311 (+ 1652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:08:25.533235 (+ 1924us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:25.533241 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:25.533256 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:25.533513 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:25.533835 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:25.533838 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:25.533840 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:25.534037 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:25.534473 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:08:25.534479 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:08:25.534791 (+ 312us) client_negotiation.cc:769] Sending connection context
0924 04:08:25.534984 (+ 193us) client_negotiation.cc:240] Negotiation successful
0924 04:08:25.535216 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":134,"threads_started":1}
I20260924 04:08:25.537380 1851 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40335
I20260924 04:08:25.537525 1851 heartbeater.cc:464] Registering TS with master...
I20260924 04:08:25.538161 1851 heartbeater.cc:511] Master 127.0.98.62:40335 requested a full tablet report, sending...
I20260924 04:08:25.539182 1508 ts_manager.cc:194] Registered new tserver with Master: c1d025e13aef436f94ce8c10ca541edf (127.0.98.2:39347)
I20260924 04:08:25.539851 1508 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:39415
I20260924 04:08:25.540477 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 1720
I20260924 04:08:25.540627 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222898807720-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:08:25.540897 392 external_mini_cluster.cc:1428] Setting key 555a21f2db45697d5d78316067fbe80d
I20260924 04:08:25.547531 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:25.539535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56543 (local address 127.0.98.62:40335)
0924 04:08:25.539712 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:25.539716 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:25.539922 (+ 206us) server_negotiation.cc:407] Connection header received
0924 04:08:25.540035 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:25.540038 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:25.540080 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:25.540164 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:25.540651 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:25.541423 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:25.542405 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:25.542647 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:25.542755 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:25.544068 (+ 1313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:25.544073 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:25.544078 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:25.544109 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:25.545907 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:25.546344 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:25.546350 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:25.546352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:25.546435 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:25.546708 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:25.546712 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:25.546714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:25.546913 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:25.547027 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:25.547284 (+ 257us) server_negotiation.cc:299] Negotiation successful
0924 04:08:25.547387 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260924 04:08:25.547550 1857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:25.539465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40335 (local address 127.0.98.2:56543)
0924 04:08:25.539835 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:25.539847 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:25.539925 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:25.540195 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:25.540197 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:25.540390 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:08:25.540523 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:25.540529 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:25.541581 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:25.541584 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:25.542287 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:25.542295 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:25.542756 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:25.542759 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:25.542846 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:25.543414 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:08:25.543432 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:08:25.543887 (+ 455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:08:25.546002 (+ 2115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:25.546006 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:25.546012 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:25.546245 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:25.546529 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:25.546532 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:25.546534 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:25.546620 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:25.547009 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:08:25.547012 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:08:25.547116 (+ 104us) client_negotiation.cc:769] Sending connection context
0924 04:08:25.547222 (+ 106us) client_negotiation.cc:240] Negotiation successful
0924 04:08:25.547364 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":133,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222898807720-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222898807720-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:08:25.568481 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222898807720-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222898807720-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222898807720-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222898807720-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40335
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222898807720-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:08:25.679360 1860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:08:25.679677 1860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:08:25.679760 1860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:08:25.683951 1860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:08:25.684056 1860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:08:25.684134 1860 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 04:08:25.684216 1860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:08:25.689670 1860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222898807720-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222898807720-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222898807720-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222898807720-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:40335
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222898807720-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:08:25.690769 1860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:08:25.691849 1860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:08:25.701179 1865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:08:25.701179 1868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:08:25.701246 1860 server_base.cc:1034] running on GCE node
W20260924 04:08:25.701179 1866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:08:25.701812 1860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:08:25.702420 1860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:08:25.703631 1860 hybrid_clock.cc:648] HybridClock initialized: now 1790222905703615 us; error 35 us; skew 500 ppm
Sep 24 04:08:25 dist-test-slave-00ld krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222905, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:08:25.706362 1860 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:08:25.707268 1860 webserver.cc:460] Webserver started at http://127.0.98.3:34599/ using document root <none> and password file <none>
I20260924 04:08:25.708003 1860 fs_manager.cc:359] Metadata directory not provided
I20260924 04:08:25.708089 1860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:08:25.708302 1860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:08:25.710790 1860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222898807720-392-0/minicluster-data/ts-2/data/instance:
uuid: "0872e2c0377340098c419c30801f9b55"
format_stamp: "Formatted at 2026-09-24 04:08:25 on dist-test-slave-00ld"
server_key: "35c6aa65b2b06d6f84b43e4712728978"
server_key_iv: "7242282bd428e18f079a68bfe4284d1c"
server_key_version: "encryptionkey@0"
I20260924 04:08:25.711360 1860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222898807720-392-0/minicluster-data/ts-2/wal/instance:
uuid: "0872e2c0377340098c419c30801f9b55"
format_stamp: "Formatted at 2026-09-24 04:08:25 on dist-test-slave-00ld"
server_key: "35c6aa65b2b06d6f84b43e4712728978"
server_key_iv: "7242282bd428e18f079a68bfe4284d1c"
server_key_version: "encryptionkey@0"
I20260924 04:08:25.715183 1860 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 04:08:25.717733 1875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:25.718935 1860 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:08:25.719120 1860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222898807720-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222898807720-392-0/minicluster-data/ts-2/wal
uuid: "0872e2c0377340098c419c30801f9b55"
format_stamp: "Formatted at 2026-09-24 04:08:25 on dist-test-slave-00ld"
server_key: "35c6aa65b2b06d6f84b43e4712728978"
server_key_iv: "7242282bd428e18f079a68bfe4284d1c"
server_key_version: "encryptionkey@0"
I20260924 04:08:25.719233 1860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222898807720-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222898807720-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222898807720-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:08:25.734202 1860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:08:25.737556 1860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:08:25.737766 1860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:08:25.739224 1860 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:08:25.739296 1860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:25.739372 1860 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:08:25.739413 1860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 04:08:25 dist-test-slave-00ld krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222905, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:08:25.749845 1860 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40375
I20260924 04:08:25.749900 1990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40375 every 8 connection(s)
I20260924 04:08:25.750885 1860 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222898807720-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:08:25.753096 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:25.739997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40935 (local address 127.0.98.62:40335)
0924 04:08:25.740161 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:25.740166 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:25.741021 (+ 855us) server_negotiation.cc:407] Connection header received
0924 04:08:25.742153 (+ 1132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:25.742158 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:25.742212 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:25.742276 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:25.744055 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:25.745026 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:25.746350 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:25.746649 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:25.746761 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:25.749489 (+ 2728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:25.749492 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:25.749495 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:25.749525 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:25.751120 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:25.751677 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:25.751680 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:25.751682 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:25.751772 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:25.752086 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:25.752088 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:25.752090 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:25.752262 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:25.752338 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:25.752879 (+ 541us) server_negotiation.cc:299] Negotiation successful
0924 04:08:25.752977 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 04:08:25.753894 1884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:25.740400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40335 (local address 127.0.98.3:40935)
0924 04:08:25.740866 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:25.740915 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:25.741886 (+ 971us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:25.742733 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:25.742744 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:25.743215 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:08:25.743889 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:25.743909 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:25.745183 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:25.745187 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:25.746222 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:25.746232 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:25.746798 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:25.746802 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:25.746974 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:25.747583 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:08:25.747598 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:08:25.749314 (+ 1716us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:08:25.751239 (+ 1925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:25.751245 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:25.751259 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:25.751583 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:25.751862 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:25.751865 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:25.751866 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:25.751996 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:25.752347 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:08:25.752352 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:08:25.752623 (+ 271us) client_negotiation.cc:769] Sending connection context
0924 04:08:25.752845 (+ 222us) client_negotiation.cc:240] Negotiation successful
0924 04:08:25.753092 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":134,"threads_started":1}
I20260924 04:08:25.754941 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 1860
I20260924 04:08:25.755069 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222898807720-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:08:25.755148 1991 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40335
I20260924 04:08:25.755281 1991 heartbeater.cc:464] Registering TS with master...
I20260924 04:08:25.755434 392 external_mini_cluster.cc:1428] Setting key 1fec804f989a4745ae9e146d3858a352
I20260924 04:08:25.755832 1991 heartbeater.cc:511] Master 127.0.98.62:40335 requested a full tablet report, sending...
I20260924 04:08:25.756922 1504 ts_manager.cc:194] Registered new tserver with Master: 0872e2c0377340098c419c30801f9b55 (127.0.98.3:40375)
I20260924 04:08:25.757681 1504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40935
I20260924 04:08:25.760017 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:08:25.766047 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:25.757325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60721 (local address 127.0.98.62:40335)
0924 04:08:25.757423 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:25.757427 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:25.758281 (+ 854us) server_negotiation.cc:407] Connection header received
0924 04:08:25.758330 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:25.758333 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:25.758376 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:25.758467 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:25.758862 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:25.759576 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:25.760607 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:25.760883 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:25.760962 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:25.762266 (+ 1304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:25.762271 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:25.762275 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:25.762304 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:25.764207 (+ 1903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:25.764789 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:25.764794 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:25.764797 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:25.764895 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:25.765207 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:25.765212 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:25.765214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:25.765452 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:25.765573 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:25.765827 (+ 254us) server_negotiation.cc:299] Negotiation successful
0924 04:08:25.765932 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 04:08:25.766180 1997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:25.757362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40335 (local address 127.0.98.3:60721)
0924 04:08:25.757616 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:25.757629 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:25.757738 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:25.758486 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:25.758489 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:25.758648 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:08:25.758750 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:25.758754 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:25.759703 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:25.759705 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:25.760492 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:25.760503 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:25.760995 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:25.761002 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:25.761130 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:25.761634 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:08:25.761654 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:08:25.762133 (+ 479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:08:25.764325 (+ 2192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:25.764328 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:25.764329 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:25.764649 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:25.765025 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:25.765028 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:25.765029 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:25.765109 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:25.765555 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:08:25.765558 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:08:25.765635 (+ 77us) client_negotiation.cc:769] Sending connection context
0924 04:08:25.765753 (+ 118us) client_negotiation.cc:240] Negotiation successful
0924 04:08:25.765915 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"mutex_wait_us":2,"thread_start_us":95,"threads_started":1}
Sep 24 04:08:25 dist-test-slave-00ld krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222905, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 04:08:25 dist-test-slave-00ld krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222905, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:08:25.787180 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:25.779280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34088 (local address 127.0.98.62:40335)
0924 04:08:25.779401 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:25.779404 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:25.779416 (+ 12us) server_negotiation.cc:407] Connection header received
0924 04:08:25.779454 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:25.779457 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:25.779492 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:25.779665 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:25.780437 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:25.781021 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:25.781910 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:25.782149 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:25.782258 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:25.784640 (+ 2382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:25.784643 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:25.784645 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:25.784667 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:25.785882 (+ 1215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:25.786245 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:25.786248 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:25.786249 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:25.786328 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:25.786547 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:25.786549 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:25.786550 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:25.786707 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:25.786801 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:25.786970 (+ 169us) server_negotiation.cc:299] Negotiation successful
0924 04:08:25.787060 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 04:08:25.790242 1508 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34088:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:08:25.793289 1508 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:08:25.807770 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:25.802429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39347 (local address 127.0.0.1:53822)
0924 04:08:25.802716 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:25.802729 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:25.802799 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:25.803808 (+ 1009us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:25.803810 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:25.803829 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:08:25.803929 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:25.803934 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:25.805269 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:25.805273 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:25.806255 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:25.806263 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:25.807308 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:25.807311 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:25.807392 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:25.807422 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 04:08:25.807571 (+ 149us) client_negotiation.cc:240] Negotiation successful
0924 04:08:25.807658 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"mutex_wait_us":5,"thread_start_us":97,"threads_started":1}
I20260924 04:08:25.807857 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:25.802573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53822 (local address 127.0.98.2:39347)
0924 04:08:25.803400 (+ 827us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:25.803412 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:25.803431 (+ 19us) server_negotiation.cc:407] Connection header received
0924 04:08:25.803510 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:25.803515 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:25.803674 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:25.803828 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:08:25.804040 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:25.805134 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:25.806371 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:25.807189 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:25.807300 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:25.807454 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:25.807619 (+ 165us) server_negotiation.cc:299] Negotiation successful
0924 04:08:25.807719 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":727,"thread_start_us":93,"threads_started":1}
I20260924 04:08:25.808349 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:25.801785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41777 (local address 127.0.0.1:53836)
0924 04:08:25.802658 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:25.802672 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:25.802799 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:25.803647 (+ 848us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:25.803651 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:25.803671 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:08:25.803795 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:25.803801 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:25.804841 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:25.804847 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:25.806345 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:25.806356 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:25.807965 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:25.807970 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:25.808071 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:25.808093 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 04:08:25.808164 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 04:08:25.808232 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":807,"mutex_wait_us":58,"thread_start_us":85,"threads_started":1}
I20260924 04:08:25.809072 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:25.801580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40375 (local address 127.0.0.1:39916)
0924 04:08:25.802358 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:25.802404 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:25.802834 (+ 430us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:25.803422 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:25.803432 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:25.803468 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:08:25.803596 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:25.803603 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:25.805051 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:25.805055 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:25.806350 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:25.806356 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:25.808642 (+ 2286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:25.808646 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:25.808765 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:25.808785 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 04:08:25.808850 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 04:08:25.808924 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"mutex_wait_us":70,"thread_start_us":95,"threads_started":1}
I20260924 04:08:25.809082 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:25.801748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39916 (local address 127.0.98.3:40375)
0924 04:08:25.802209 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:25.802222 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:25.802942 (+ 720us) server_negotiation.cc:407] Connection header received
0924 04:08:25.803040 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:25.803046 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:25.803261 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:25.803417 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:08:25.803714 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:25.804933 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:25.807498 (+ 2565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:25.808287 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:25.808626 (+ 339us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:25.808821 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:25.808871 (+ 50us) server_negotiation.cc:299] Negotiation successful
0924 04:08:25.808946 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":103,"threads_started":1}
I20260924 04:08:25.810581 1785 tablet_service.cc:1467] Processing CreateTablet for tablet 349d0b8255ec4badbd0e33c5f56e6d9d (DEFAULT_TABLE table=test-table [id=2faa3a339f784d67b621d2b1172460d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:08:25.811336 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:25.802365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53836 (local address 127.0.98.1:41777)
0924 04:08:25.803084 (+ 719us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:25.803092 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:25.803116 (+ 24us) server_negotiation.cc:407] Connection header received
0924 04:08:25.803196 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:25.803203 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:25.803386 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:25.803524 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:08:25.803917 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:25.804669 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:25.806465 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:25.807189 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:25.807300 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:25.807439 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:25.811086 (+ 3647us) server_negotiation.cc:299] Negotiation successful
0924 04:08:25.811180 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":68,"threads_started":1}
I20260924 04:08:25.811256 1925 tablet_service.cc:1467] Processing CreateTablet for tablet 349d0b8255ec4badbd0e33c5f56e6d9d (DEFAULT_TABLE table=test-table [id=2faa3a339f784d67b621d2b1172460d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:08:25.811650 1785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 349d0b8255ec4badbd0e33c5f56e6d9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:08:25.812243 1925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 349d0b8255ec4badbd0e33c5f56e6d9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:08:25.813758 1645 tablet_service.cc:1467] Processing CreateTablet for tablet 349d0b8255ec4badbd0e33c5f56e6d9d (DEFAULT_TABLE table=test-table [id=2faa3a339f784d67b621d2b1172460d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:08:25.814654 1645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 349d0b8255ec4badbd0e33c5f56e6d9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:08:25.818141 2014 tablet_bootstrap.cc:492] T 349d0b8255ec4badbd0e33c5f56e6d9d P c1d025e13aef436f94ce8c10ca541edf: Bootstrap starting.
I20260924 04:08:25.819558 2015 tablet_bootstrap.cc:492] T 349d0b8255ec4badbd0e33c5f56e6d9d P 0872e2c0377340098c419c30801f9b55: Bootstrap starting.
I20260924 04:08:25.820513 2014 tablet_bootstrap.cc:654] T 349d0b8255ec4badbd0e33c5f56e6d9d P c1d025e13aef436f94ce8c10ca541edf: Neither blocks nor log segments found. Creating new log.
I20260924 04:08:25.821375 2014 log.cc:824] T 349d0b8255ec4badbd0e33c5f56e6d9d P c1d025e13aef436f94ce8c10ca541edf: Log is configured to *not* fsync() on all Append() calls
I20260924 04:08:25.821385 2016 tablet_bootstrap.cc:492] T 349d0b8255ec4badbd0e33c5f56e6d9d P 7db0dad5cd924a5f80a51795d2a0315d: Bootstrap starting.
I20260924 04:08:25.821700 2015 tablet_bootstrap.cc:654] T 349d0b8255ec4badbd0e33c5f56e6d9d P 0872e2c0377340098c419c30801f9b55: Neither blocks nor log segments found. Creating new log.
I20260924 04:08:25.822389 2015 log.cc:824] T 349d0b8255ec4badbd0e33c5f56e6d9d P 0872e2c0377340098c419c30801f9b55: Log is configured to *not* fsync() on all Append() calls
I20260924 04:08:25.823645 2016 tablet_bootstrap.cc:654] T 349d0b8255ec4badbd0e33c5f56e6d9d P 7db0dad5cd924a5f80a51795d2a0315d: Neither blocks nor log segments found. Creating new log.
I20260924 04:08:25.823992 2014 tablet_bootstrap.cc:492] T 349d0b8255ec4badbd0e33c5f56e6d9d P c1d025e13aef436f94ce8c10ca541edf: No bootstrap required, opened a new log
I20260924 04:08:25.824179 2014 ts_tablet_manager.cc:1397] T 349d0b8255ec4badbd0e33c5f56e6d9d P c1d025e13aef436f94ce8c10ca541edf: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 04:08:25.824270 2015 tablet_bootstrap.cc:492] T 349d0b8255ec4badbd0e33c5f56e6d9d P 0872e2c0377340098c419c30801f9b55: No bootstrap required, opened a new log
I20260924 04:08:25.824362 2016 log.cc:824] T 349d0b8255ec4badbd0e33c5f56e6d9d P 7db0dad5cd924a5f80a51795d2a0315d: Log is configured to *not* fsync() on all Append() calls
I20260924 04:08:25.824440 2015 ts_tablet_manager.cc:1397] T 349d0b8255ec4badbd0e33c5f56e6d9d P 0872e2c0377340098c419c30801f9b55: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 04:08:25.825846 2016 tablet_bootstrap.cc:492] T 349d0b8255ec4badbd0e33c5f56e6d9d P 7db0dad5cd924a5f80a51795d2a0315d: No bootstrap required, opened a new log
I20260924 04:08:25.826035 2016 ts_tablet_manager.cc:1397] T 349d0b8255ec4badbd0e33c5f56e6d9d P 7db0dad5cd924a5f80a51795d2a0315d: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 04:08:25.828114 2014 raft_consensus.cc:348] T 349d0b8255ec4badbd0e33c5f56e6d9d P c1d025e13aef436f94ce8c10ca541edf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0872e2c0377340098c419c30801f9b55" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40375 } } peers { permanent_uuid: "7db0dad5cd924a5f80a51795d2a0315d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41777 } } peers { permanent_uuid: "c1d025e13aef436f94ce8c10ca541edf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39347 } }
I20260924 04:08:25.828114 2015 raft_consensus.cc:348] T 349d0b8255ec4badbd0e33c5f56e6d9d P 0872e2c0377340098c419c30801f9b55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0872e2c0377340098c419c30801f9b55" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40375 } } peers { permanent_uuid: "7db0dad5cd924a5f80a51795d2a0315d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41777 } } peers { permanent_uuid: "c1d025e13aef436f94ce8c10ca541edf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39347 } }
I20260924 04:08:25.828374 2014 raft_consensus.cc:374] T 349d0b8255ec4badbd0e33c5f56e6d9d P c1d025e13aef436f94ce8c10ca541edf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:08:25.828374 2015 raft_consensus.cc:374] T 349d0b8255ec4badbd0e33c5f56e6d9d P 0872e2c0377340098c419c30801f9b55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:08:25.828433 2014 raft_consensus.cc:729] T 349d0b8255ec4badbd0e33c5f56e6d9d P c1d025e13aef436f94ce8c10ca541edf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1d025e13aef436f94ce8c10ca541edf, State: Initialized, Role: FOLLOWER
I20260924 04:08:25.828441 2015 raft_consensus.cc:729] T 349d0b8255ec4badbd0e33c5f56e6d9d P 0872e2c0377340098c419c30801f9b55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0872e2c0377340098c419c30801f9b55, State: Initialized, Role: FOLLOWER
I20260924 04:08:25.828924 2014 consensus_queue.cc:260] T 349d0b8255ec4badbd0e33c5f56e6d9d P c1d025e13aef436f94ce8c10ca541edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0872e2c0377340098c419c30801f9b55" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40375 } } peers { permanent_uuid: "7db0dad5cd924a5f80a51795d2a0315d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41777 } } peers { permanent_uuid: "c1d025e13aef436f94ce8c10ca541edf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39347 } }
I20260924 04:08:25.828924 2015 consensus_queue.cc:260] T 349d0b8255ec4badbd0e33c5f56e6d9d P 0872e2c0377340098c419c30801f9b55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0872e2c0377340098c419c30801f9b55" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40375 } } peers { permanent_uuid: "7db0dad5cd924a5f80a51795d2a0315d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41777 } } peers { permanent_uuid: "c1d025e13aef436f94ce8c10ca541edf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39347 } }
I20260924 04:08:25.829602 1851 heartbeater.cc:503] Master 127.0.98.62:40335 was elected leader, sending a full tablet report...
I20260924 04:08:25.829484 2016 raft_consensus.cc:348] T 349d0b8255ec4badbd0e33c5f56e6d9d P 7db0dad5cd924a5f80a51795d2a0315d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0872e2c0377340098c419c30801f9b55" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40375 } } peers { permanent_uuid: "7db0dad5cd924a5f80a51795d2a0315d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41777 } } peers { permanent_uuid: "c1d025e13aef436f94ce8c10ca541edf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39347 } }
I20260924 04:08:25.829744 2016 raft_consensus.cc:374] T 349d0b8255ec4badbd0e33c5f56e6d9d P 7db0dad5cd924a5f80a51795d2a0315d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:08:25.829809 2016 raft_consensus.cc:729] T 349d0b8255ec4badbd0e33c5f56e6d9d P 7db0dad5cd924a5f80a51795d2a0315d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7db0dad5cd924a5f80a51795d2a0315d, State: Initialized, Role: FOLLOWER
I20260924 04:08:25.829929 1991 heartbeater.cc:503] Master 127.0.98.62:40335 was elected leader, sending a full tablet report...
I20260924 04:08:25.830262 2016 consensus_queue.cc:260] T 349d0b8255ec4badbd0e33c5f56e6d9d P 7db0dad5cd924a5f80a51795d2a0315d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0872e2c0377340098c419c30801f9b55" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40375 } } peers { permanent_uuid: "7db0dad5cd924a5f80a51795d2a0315d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41777 } } peers { permanent_uuid: "c1d025e13aef436f94ce8c10ca541edf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39347 } }
I20260924 04:08:25.830912 2015 ts_tablet_manager.cc:1428] T 349d0b8255ec4badbd0e33c5f56e6d9d P 0872e2c0377340098c419c30801f9b55: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 04:08:25.831384 2014 ts_tablet_manager.cc:1428] T 349d0b8255ec4badbd0e33c5f56e6d9d P c1d025e13aef436f94ce8c10ca541edf: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260924 04:08:25.831904 2016 ts_tablet_manager.cc:1428] T 349d0b8255ec4badbd0e33c5f56e6d9d P 7db0dad5cd924a5f80a51795d2a0315d: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 04:08:25.832000 1711 heartbeater.cc:503] Master 127.0.98.62:40335 was elected leader, sending a full tablet report...
W20260924 04:08:26.001894 1992 tablet.cc:2369] T 349d0b8255ec4badbd0e33c5f56e6d9d P 0872e2c0377340098c419c30801f9b55: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 04:08:26.034705 1852 tablet.cc:2369] T 349d0b8255ec4badbd0e33c5f56e6d9d P c1d025e13aef436f94ce8c10ca541edf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 04:08:26.038174 1712 tablet.cc:2369] T 349d0b8255ec4badbd0e33c5f56e6d9d P 7db0dad5cd924a5f80a51795d2a0315d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:08:26.046614 2020 raft_consensus.cc:482] T 349d0b8255ec4badbd0e33c5f56e6d9d P c1d025e13aef436f94ce8c10ca541edf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:08:26.046876 2020 raft_consensus.cc:504] T 349d0b8255ec4badbd0e33c5f56e6d9d P c1d025e13aef436f94ce8c10ca541edf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0872e2c0377340098c419c30801f9b55" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40375 } } peers { permanent_uuid: "7db0dad5cd924a5f80a51795d2a0315d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41777 } } peers { permanent_uuid: "c1d025e13aef436f94ce8c10ca541edf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39347 } }
I20260924 04:08:26.048125 2020 leader_election.cc:288] T 349d0b8255ec4badbd0e33c5f56e6d9d P c1d025e13aef436f94ce8c10ca541edf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0872e2c0377340098c419c30801f9b55 (127.0.98.3:40375), 7db0dad5cd924a5f80a51795d2a0315d (127.0.98.1:41777)
I20260924 04:08:26.051846 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:26.048467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53789 (local address 127.0.98.3:40375)
0924 04:08:26.048652 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:26.048658 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:26.048776 (+ 118us) server_negotiation.cc:407] Connection header received
0924 04:08:26.048934 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:26.048942 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:26.048983 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:26.049069 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:08:26.049336 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:26.049845 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:26.050909 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:26.051305 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:26.051418 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:26.051482 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:26.051655 (+ 173us) server_negotiation.cc:299] Negotiation successful
0924 04:08:26.051722 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260924 04:08:26.051867 2023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:26.048333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40375 (local address 127.0.98.2:53789)
0924 04:08:26.048646 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:26.048663 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:26.048814 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:26.049082 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:26.049084 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:26.049099 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:08:26.049228 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:26.049234 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:26.049955 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:26.049961 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:26.050781 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:26.050790 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:26.051423 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:26.051427 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:26.051523 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:26.051552 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 04:08:26.051640 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 04:08:26.051714 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"mutex_wait_us":2,"thread_start_us":94,"threads_started":1}
I20260924 04:08:26.052074 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:26.048564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36773 (local address 127.0.98.1:41777)
0924 04:08:26.048715 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:26.048719 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:26.048858 (+ 139us) server_negotiation.cc:407] Connection header received
0924 04:08:26.049013 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:26.049021 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:26.049062 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:26.049145 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:08:26.049404 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:26.049919 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:26.051115 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:26.051532 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:26.051646 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:26.051715 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:26.051890 (+ 175us) server_negotiation.cc:299] Negotiation successful
0924 04:08:26.051960 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 04:08:26.052093 2024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:26.048408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41777 (local address 127.0.98.2:36773)
0924 04:08:26.048775 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:26.048786 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:26.048903 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:26.049154 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:26.049157 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:26.049171 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:08:26.049294 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:26.049300 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:26.050046 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:26.050049 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:26.050988 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:26.050997 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:26.051648 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:26.051652 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:26.051763 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:26.051783 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 04:08:26.051876 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 04:08:26.051944 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"mutex_wait_us":22,"thread_start_us":78,"threads_started":1}
I20260924 04:08:26.052902 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "349d0b8255ec4badbd0e33c5f56e6d9d" candidate_uuid: "c1d025e13aef436f94ce8c10ca541edf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0872e2c0377340098c419c30801f9b55" is_pre_election: true
I20260924 04:08:26.052950 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "349d0b8255ec4badbd0e33c5f56e6d9d" candidate_uuid: "c1d025e13aef436f94ce8c10ca541edf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7db0dad5cd924a5f80a51795d2a0315d" is_pre_election: true
I20260924 04:08:26.053213 1945 raft_consensus.cc:2454] T 349d0b8255ec4badbd0e33c5f56e6d9d P 0872e2c0377340098c419c30801f9b55 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1d025e13aef436f94ce8c10ca541edf in term 0.
I20260924 04:08:26.053259 1663 raft_consensus.cc:2454] T 349d0b8255ec4badbd0e33c5f56e6d9d P 7db0dad5cd924a5f80a51795d2a0315d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1d025e13aef436f94ce8c10ca541edf in term 0.
I20260924 04:08:26.053707 1738 leader_election.cc:302] T 349d0b8255ec4badbd0e33c5f56e6d9d P c1d025e13aef436f94ce8c10ca541edf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0872e2c0377340098c419c30801f9b55, 7db0dad5cd924a5f80a51795d2a0315d, c1d025e13aef436f94ce8c10ca541edf; no voters:
I20260924 04:08:26.053975 2020 raft_consensus.cc:2781] T 349d0b8255ec4badbd0e33c5f56e6d9d P c1d025e13aef436f94ce8c10ca541edf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:08:26.054067 2020 raft_consensus.cc:482] T 349d0b8255ec4badbd0e33c5f56e6d9d P c1d025e13aef436f94ce8c10ca541edf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:08:26.054122 2020 raft_consensus.cc:3037] T 349d0b8255ec4badbd0e33c5f56e6d9d P c1d025e13aef436f94ce8c10ca541edf [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:08:26.055219 2020 raft_consensus.cc:504] T 349d0b8255ec4badbd0e33c5f56e6d9d P c1d025e13aef436f94ce8c10ca541edf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0872e2c0377340098c419c30801f9b55" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40375 } } peers { permanent_uuid: "7db0dad5cd924a5f80a51795d2a0315d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41777 } } peers { permanent_uuid: "c1d025e13aef436f94ce8c10ca541edf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39347 } }
I20260924 04:08:26.055627 2020 leader_election.cc:288] T 349d0b8255ec4badbd0e33c5f56e6d9d P c1d025e13aef436f94ce8c10ca541edf [CANDIDATE]: Term 1 election: Requested vote from peers 0872e2c0377340098c419c30801f9b55 (127.0.98.3:40375), 7db0dad5cd924a5f80a51795d2a0315d (127.0.98.1:41777)
I20260924 04:08:26.056003 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "349d0b8255ec4badbd0e33c5f56e6d9d" candidate_uuid: "c1d025e13aef436f94ce8c10ca541edf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7db0dad5cd924a5f80a51795d2a0315d"
I20260924 04:08:26.056003 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "349d0b8255ec4badbd0e33c5f56e6d9d" candidate_uuid: "c1d025e13aef436f94ce8c10ca541edf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0872e2c0377340098c419c30801f9b55"
I20260924 04:08:26.056141 1945 raft_consensus.cc:3037] T 349d0b8255ec4badbd0e33c5f56e6d9d P 0872e2c0377340098c419c30801f9b55 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:08:26.056141 1663 raft_consensus.cc:3037] T 349d0b8255ec4badbd0e33c5f56e6d9d P 7db0dad5cd924a5f80a51795d2a0315d [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:08:26.057560 1663 raft_consensus.cc:2454] T 349d0b8255ec4badbd0e33c5f56e6d9d P 7db0dad5cd924a5f80a51795d2a0315d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1d025e13aef436f94ce8c10ca541edf in term 1.
I20260924 04:08:26.057914 1737 leader_election.cc:302] T 349d0b8255ec4badbd0e33c5f56e6d9d P c1d025e13aef436f94ce8c10ca541edf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7db0dad5cd924a5f80a51795d2a0315d, c1d025e13aef436f94ce8c10ca541edf; no voters:
I20260924 04:08:26.057931 1945 raft_consensus.cc:2454] T 349d0b8255ec4badbd0e33c5f56e6d9d P 0872e2c0377340098c419c30801f9b55 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1d025e13aef436f94ce8c10ca541edf in term 1.
I20260924 04:08:26.058103 2020 raft_consensus.cc:2781] T 349d0b8255ec4badbd0e33c5f56e6d9d P c1d025e13aef436f94ce8c10ca541edf [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:08:26.058288 2020 raft_consensus.cc:686] T 349d0b8255ec4badbd0e33c5f56e6d9d P c1d025e13aef436f94ce8c10ca541edf [term 1 LEADER]: Becoming Leader. State: Replica: c1d025e13aef436f94ce8c10ca541edf, State: Running, Role: LEADER
I20260924 04:08:26.058702 2020 consensus_queue.cc:237] T 349d0b8255ec4badbd0e33c5f56e6d9d P c1d025e13aef436f94ce8c10ca541edf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0872e2c0377340098c419c30801f9b55" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40375 } } peers { permanent_uuid: "7db0dad5cd924a5f80a51795d2a0315d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41777 } } peers { permanent_uuid: "c1d025e13aef436f94ce8c10ca541edf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39347 } }
I20260924 04:08:26.061808 1507 catalog_manager.cc:5495] T 349d0b8255ec4badbd0e33c5f56e6d9d P c1d025e13aef436f94ce8c10ca541edf reported cstate change: term changed from 0 to 1, leader changed from <none> to c1d025e13aef436f94ce8c10ca541edf (127.0.98.2). New cstate: current_term: 1 leader_uuid: "c1d025e13aef436f94ce8c10ca541edf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0872e2c0377340098c419c30801f9b55" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40375 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7db0dad5cd924a5f80a51795d2a0315d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41777 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1d025e13aef436f94ce8c10ca541edf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39347 } health_report { overall_health: HEALTHY } } }
I20260924 04:08:26.104823 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:26.101014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53834 (local address 127.0.98.2:39347)
0924 04:08:26.101183 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:26.101187 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:26.101202 (+ 15us) server_negotiation.cc:407] Connection header received
0924 04:08:26.101432 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:26.101434 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:26.101487 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:26.101594 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:08:26.101860 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:26.102370 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:26.103289 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:26.103520 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:26.103633 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:26.103967 (+ 334us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:08:26.104362 (+ 395us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:08:26.104487 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:26.104597 (+ 110us) server_negotiation.cc:299] Negotiation successful
0924 04:08:26.104653 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 04:08:26.110611 1663 raft_consensus.cc:1261] T 349d0b8255ec4badbd0e33c5f56e6d9d P 7db0dad5cd924a5f80a51795d2a0315d [term 1 FOLLOWER]: Refusing update from remote peer c1d025e13aef436f94ce8c10ca541edf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:08:26.110611 1945 raft_consensus.cc:1261] T 349d0b8255ec4badbd0e33c5f56e6d9d P 0872e2c0377340098c419c30801f9b55 [term 1 FOLLOWER]: Refusing update from remote peer c1d025e13aef436f94ce8c10ca541edf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:08:26.111333 2020 consensus_queue.cc:1035] T 349d0b8255ec4badbd0e33c5f56e6d9d P c1d025e13aef436f94ce8c10ca541edf [LEADER]: Connected to new peer: Peer: permanent_uuid: "7db0dad5cd924a5f80a51795d2a0315d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:08:26.111578 2025 consensus_queue.cc:1035] T 349d0b8255ec4badbd0e33c5f56e6d9d P c1d025e13aef436f94ce8c10ca541edf [LEADER]: Connected to new peer: Peer: permanent_uuid: "0872e2c0377340098c419c30801f9b55" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:08:26.118844 2028 mvcc.cc:205] Tried to move back new op lower bound from 7332753023421251584 to 7332753023218794496. Current Snapshot: MvccSnapshot[applied={T|T < 7332753023421251584}]
I20260924 04:08:26.132761 1508 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34088:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:08:26.132957 1508 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34088:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:08:26.135234 1508 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a9550222f0fa4b488486b4d4ff95b688: Sending DeleteTablet for 3 replicas of tablet 349d0b8255ec4badbd0e33c5f56e6d9d
I20260924 04:08:26.135988 1785 tablet_service.cc:1514] Processing DeleteTablet for tablet 349d0b8255ec4badbd0e33c5f56e6d9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:08:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53822
I20260924 04:08:26.135989 1925 tablet_service.cc:1514] Processing DeleteTablet for tablet 349d0b8255ec4badbd0e33c5f56e6d9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:08:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39916
I20260924 04:08:26.136047 1645 tablet_service.cc:1514] Processing DeleteTablet for tablet 349d0b8255ec4badbd0e33c5f56e6d9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:08:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53836
I20260924 04:08:26.136574 2038 tablet_replica.cc:330] T 349d0b8255ec4badbd0e33c5f56e6d9d P 0872e2c0377340098c419c30801f9b55: stopping tablet replica
I20260924 04:08:26.136639 2040 tablet_replica.cc:330] T 349d0b8255ec4badbd0e33c5f56e6d9d P 7db0dad5cd924a5f80a51795d2a0315d: stopping tablet replica
I20260924 04:08:26.136822 2039 tablet_replica.cc:330] T 349d0b8255ec4badbd0e33c5f56e6d9d P c1d025e13aef436f94ce8c10ca541edf: stopping tablet replica
I20260924 04:08:26.136988 2040 raft_consensus.cc:2229] T 349d0b8255ec4badbd0e33c5f56e6d9d P 7db0dad5cd924a5f80a51795d2a0315d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:08:26.137295 2038 raft_consensus.cc:2229] T 349d0b8255ec4badbd0e33c5f56e6d9d P 0872e2c0377340098c419c30801f9b55 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:08:26.137322 2040 raft_consensus.cc:2258] T 349d0b8255ec4badbd0e33c5f56e6d9d P 7db0dad5cd924a5f80a51795d2a0315d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:08:26.137403 2039 raft_consensus.cc:2229] T 349d0b8255ec4badbd0e33c5f56e6d9d P c1d025e13aef436f94ce8c10ca541edf [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:08:26.137859 2039 raft_consensus.cc:2258] T 349d0b8255ec4badbd0e33c5f56e6d9d P c1d025e13aef436f94ce8c10ca541edf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:08:26.138111 1508 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34088:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 04:08:26.138530 2038 raft_consensus.cc:2258] T 349d0b8255ec4badbd0e33c5f56e6d9d P 0872e2c0377340098c419c30801f9b55 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260924 04:08:26.138540 1508 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:08:26.139639 2039 ts_tablet_manager.cc:1905] T 349d0b8255ec4badbd0e33c5f56e6d9d P c1d025e13aef436f94ce8c10ca541edf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:08:26.140723 2038 ts_tablet_manager.cc:1905] T 349d0b8255ec4badbd0e33c5f56e6d9d P 0872e2c0377340098c419c30801f9b55: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:08:26.142522 2039 ts_tablet_manager.cc:1918] T 349d0b8255ec4badbd0e33c5f56e6d9d P c1d025e13aef436f94ce8c10ca541edf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:08:26.142617 2039 log.cc:1196] T 349d0b8255ec4badbd0e33c5f56e6d9d P c1d025e13aef436f94ce8c10ca541edf: Deleting WAL directory at /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222898807720-392-0/minicluster-data/ts-1/wal/wals/349d0b8255ec4badbd0e33c5f56e6d9d
I20260924 04:08:26.142913 2039 ts_tablet_manager.cc:1939] T 349d0b8255ec4badbd0e33c5f56e6d9d P c1d025e13aef436f94ce8c10ca541edf: Deleting consensus metadata
I20260924 04:08:26.143834 1785 tablet_service.cc:1467] Processing CreateTablet for tablet 7b9b6c8f72b44244883984e3b3d67754 (DEFAULT_TABLE table=test-table [id=2a863ed1a75d4d87b331c07416fce0dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:08:26.143919 1925 tablet_service.cc:1467] Processing CreateTablet for tablet 7b9b6c8f72b44244883984e3b3d67754 (DEFAULT_TABLE table=test-table [id=2a863ed1a75d4d87b331c07416fce0dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:08:26.144119 1785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b9b6c8f72b44244883984e3b3d67754. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:08:26.144182 1925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b9b6c8f72b44244883984e3b3d67754. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:08:26.144666 2040 ts_tablet_manager.cc:1905] T 349d0b8255ec4badbd0e33c5f56e6d9d P 7db0dad5cd924a5f80a51795d2a0315d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:08:26.145334 1645 tablet_service.cc:1467] Processing CreateTablet for tablet 7b9b6c8f72b44244883984e3b3d67754 (DEFAULT_TABLE table=test-table [id=2a863ed1a75d4d87b331c07416fce0dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:08:26.145572 1645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b9b6c8f72b44244883984e3b3d67754. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:08:26.146492 2015 tablet_bootstrap.cc:492] T 7b9b6c8f72b44244883984e3b3d67754 P 0872e2c0377340098c419c30801f9b55: Bootstrap starting.
I20260924 04:08:26.147359 2040 ts_tablet_manager.cc:1918] T 349d0b8255ec4badbd0e33c5f56e6d9d P 7db0dad5cd924a5f80a51795d2a0315d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:08:26.147454 2040 log.cc:1196] T 349d0b8255ec4badbd0e33c5f56e6d9d P 7db0dad5cd924a5f80a51795d2a0315d: Deleting WAL directory at /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222898807720-392-0/minicluster-data/ts-0/wal/wals/349d0b8255ec4badbd0e33c5f56e6d9d
I20260924 04:08:26.147493 2015 tablet_bootstrap.cc:654] T 7b9b6c8f72b44244883984e3b3d67754 P 0872e2c0377340098c419c30801f9b55: Neither blocks nor log segments found. Creating new log.
I20260924 04:08:26.147766 2040 ts_tablet_manager.cc:1939] T 349d0b8255ec4badbd0e33c5f56e6d9d P 7db0dad5cd924a5f80a51795d2a0315d: Deleting consensus metadata
I20260924 04:08:26.148132 2016 tablet_bootstrap.cc:492] T 7b9b6c8f72b44244883984e3b3d67754 P 7db0dad5cd924a5f80a51795d2a0315d: Bootstrap starting.
I20260924 04:08:26.149053 2016 tablet_bootstrap.cc:654] T 7b9b6c8f72b44244883984e3b3d67754 P 7db0dad5cd924a5f80a51795d2a0315d: Neither blocks nor log segments found. Creating new log.
I20260924 04:08:26.149237 1493 catalog_manager.cc:4841] TS 7db0dad5cd924a5f80a51795d2a0315d (127.0.98.1:41777): tablet 349d0b8255ec4badbd0e33c5f56e6d9d (table test-table [id=2faa3a339f784d67b621d2b1172460d5]) successfully deleted
I20260924 04:08:26.149423 1495 catalog_manager.cc:4841] TS c1d025e13aef436f94ce8c10ca541edf (127.0.98.2:39347): tablet 349d0b8255ec4badbd0e33c5f56e6d9d (table test-table [id=2faa3a339f784d67b621d2b1172460d5]) successfully deleted
I20260924 04:08:26.150641 2014 tablet_bootstrap.cc:492] T 7b9b6c8f72b44244883984e3b3d67754 P c1d025e13aef436f94ce8c10ca541edf: Bootstrap starting.
I20260924 04:08:26.151110 2038 ts_tablet_manager.cc:1918] T 349d0b8255ec4badbd0e33c5f56e6d9d P 0872e2c0377340098c419c30801f9b55: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:08:26.151217 2038 log.cc:1196] T 349d0b8255ec4badbd0e33c5f56e6d9d P 0872e2c0377340098c419c30801f9b55: Deleting WAL directory at /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222898807720-392-0/minicluster-data/ts-2/wal/wals/349d0b8255ec4badbd0e33c5f56e6d9d
I20260924 04:08:26.151535 2038 ts_tablet_manager.cc:1939] T 349d0b8255ec4badbd0e33c5f56e6d9d P 0872e2c0377340098c419c30801f9b55: Deleting consensus metadata
I20260924 04:08:26.151744 2014 tablet_bootstrap.cc:654] T 7b9b6c8f72b44244883984e3b3d67754 P c1d025e13aef436f94ce8c10ca541edf: Neither blocks nor log segments found. Creating new log.
I20260924 04:08:26.152457 2016 tablet_bootstrap.cc:492] T 7b9b6c8f72b44244883984e3b3d67754 P 7db0dad5cd924a5f80a51795d2a0315d: No bootstrap required, opened a new log
I20260924 04:08:26.152573 2016 ts_tablet_manager.cc:1397] T 7b9b6c8f72b44244883984e3b3d67754 P 7db0dad5cd924a5f80a51795d2a0315d: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260924 04:08:26.153047 2014 tablet_bootstrap.cc:492] T 7b9b6c8f72b44244883984e3b3d67754 P c1d025e13aef436f94ce8c10ca541edf: No bootstrap required, opened a new log
I20260924 04:08:26.153133 2014 ts_tablet_manager.cc:1397] T 7b9b6c8f72b44244883984e3b3d67754 P c1d025e13aef436f94ce8c10ca541edf: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 04:08:26.153312 2016 raft_consensus.cc:348] T 7b9b6c8f72b44244883984e3b3d67754 P 7db0dad5cd924a5f80a51795d2a0315d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db0dad5cd924a5f80a51795d2a0315d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41777 } } peers { permanent_uuid: "c1d025e13aef436f94ce8c10ca541edf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39347 } } peers { permanent_uuid: "0872e2c0377340098c419c30801f9b55" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40375 } }
I20260924 04:08:26.153431 2016 raft_consensus.cc:374] T 7b9b6c8f72b44244883984e3b3d67754 P 7db0dad5cd924a5f80a51795d2a0315d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:08:26.153467 2016 raft_consensus.cc:729] T 7b9b6c8f72b44244883984e3b3d67754 P 7db0dad5cd924a5f80a51795d2a0315d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7db0dad5cd924a5f80a51795d2a0315d, State: Initialized, Role: FOLLOWER
I20260924 04:08:26.153573 2014 raft_consensus.cc:348] T 7b9b6c8f72b44244883984e3b3d67754 P c1d025e13aef436f94ce8c10ca541edf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db0dad5cd924a5f80a51795d2a0315d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41777 } } peers { permanent_uuid: "c1d025e13aef436f94ce8c10ca541edf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39347 } } peers { permanent_uuid: "0872e2c0377340098c419c30801f9b55" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40375 } }
I20260924 04:08:26.153612 2016 consensus_queue.cc:260] T 7b9b6c8f72b44244883984e3b3d67754 P 7db0dad5cd924a5f80a51795d2a0315d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db0dad5cd924a5f80a51795d2a0315d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41777 } } peers { permanent_uuid: "c1d025e13aef436f94ce8c10ca541edf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39347 } } peers { permanent_uuid: "0872e2c0377340098c419c30801f9b55" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40375 } }
I20260924 04:08:26.153667 2014 raft_consensus.cc:374] T 7b9b6c8f72b44244883984e3b3d67754 P c1d025e13aef436f94ce8c10ca541edf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:08:26.153699 2014 raft_consensus.cc:729] T 7b9b6c8f72b44244883984e3b3d67754 P c1d025e13aef436f94ce8c10ca541edf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1d025e13aef436f94ce8c10ca541edf, State: Initialized, Role: FOLLOWER
I20260924 04:08:26.153810 2014 consensus_queue.cc:260] T 7b9b6c8f72b44244883984e3b3d67754 P c1d025e13aef436f94ce8c10ca541edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db0dad5cd924a5f80a51795d2a0315d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41777 } } peers { permanent_uuid: "c1d025e13aef436f94ce8c10ca541edf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39347 } } peers { permanent_uuid: "0872e2c0377340098c419c30801f9b55" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40375 } }
I20260924 04:08:26.153880 2016 ts_tablet_manager.cc:1428] T 7b9b6c8f72b44244883984e3b3d67754 P 7db0dad5cd924a5f80a51795d2a0315d: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260924 04:08:26.154059 2014 ts_tablet_manager.cc:1428] T 7b9b6c8f72b44244883984e3b3d67754 P c1d025e13aef436f94ce8c10ca541edf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:08:26.154979 1494 catalog_manager.cc:4841] TS 0872e2c0377340098c419c30801f9b55 (127.0.98.3:40375): tablet 349d0b8255ec4badbd0e33c5f56e6d9d (table test-table [id=2faa3a339f784d67b621d2b1172460d5]) successfully deleted
I20260924 04:08:26.155552 2015 tablet_bootstrap.cc:492] T 7b9b6c8f72b44244883984e3b3d67754 P 0872e2c0377340098c419c30801f9b55: No bootstrap required, opened a new log
I20260924 04:08:26.155639 2015 ts_tablet_manager.cc:1397] T 7b9b6c8f72b44244883984e3b3d67754 P 0872e2c0377340098c419c30801f9b55: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260924 04:08:26.156117 2015 raft_consensus.cc:348] T 7b9b6c8f72b44244883984e3b3d67754 P 0872e2c0377340098c419c30801f9b55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db0dad5cd924a5f80a51795d2a0315d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41777 } } peers { permanent_uuid: "c1d025e13aef436f94ce8c10ca541edf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39347 } } peers { permanent_uuid: "0872e2c0377340098c419c30801f9b55" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40375 } }
I20260924 04:08:26.156240 2015 raft_consensus.cc:374] T 7b9b6c8f72b44244883984e3b3d67754 P 0872e2c0377340098c419c30801f9b55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:08:26.156278 2015 raft_consensus.cc:729] T 7b9b6c8f72b44244883984e3b3d67754 P 0872e2c0377340098c419c30801f9b55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0872e2c0377340098c419c30801f9b55, State: Initialized, Role: FOLLOWER
I20260924 04:08:26.156397 2015 consensus_queue.cc:260] T 7b9b6c8f72b44244883984e3b3d67754 P 0872e2c0377340098c419c30801f9b55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db0dad5cd924a5f80a51795d2a0315d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41777 } } peers { permanent_uuid: "c1d025e13aef436f94ce8c10ca541edf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39347 } } peers { permanent_uuid: "0872e2c0377340098c419c30801f9b55" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40375 } }
I20260924 04:08:26.156708 2015 ts_tablet_manager.cc:1428] T 7b9b6c8f72b44244883984e3b3d67754 P 0872e2c0377340098c419c30801f9b55: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260924 04:08:26.269853 2035 raft_consensus.cc:482] T 7b9b6c8f72b44244883984e3b3d67754 P c1d025e13aef436f94ce8c10ca541edf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:08:26.270011 2035 raft_consensus.cc:504] T 7b9b6c8f72b44244883984e3b3d67754 P c1d025e13aef436f94ce8c10ca541edf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db0dad5cd924a5f80a51795d2a0315d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41777 } } peers { permanent_uuid: "c1d025e13aef436f94ce8c10ca541edf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39347 } } peers { permanent_uuid: "0872e2c0377340098c419c30801f9b55" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40375 } }
I20260924 04:08:26.270480 2035 leader_election.cc:288] T 7b9b6c8f72b44244883984e3b3d67754 P c1d025e13aef436f94ce8c10ca541edf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7db0dad5cd924a5f80a51795d2a0315d (127.0.98.1:41777), 0872e2c0377340098c419c30801f9b55 (127.0.98.3:40375)
I20260924 04:08:26.271019 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b9b6c8f72b44244883984e3b3d67754" candidate_uuid: "c1d025e13aef436f94ce8c10ca541edf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0872e2c0377340098c419c30801f9b55" is_pre_election: true
I20260924 04:08:26.271054 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b9b6c8f72b44244883984e3b3d67754" candidate_uuid: "c1d025e13aef436f94ce8c10ca541edf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7db0dad5cd924a5f80a51795d2a0315d" is_pre_election: true
I20260924 04:08:26.271214 1663 raft_consensus.cc:2454] T 7b9b6c8f72b44244883984e3b3d67754 P 7db0dad5cd924a5f80a51795d2a0315d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1d025e13aef436f94ce8c10ca541edf in term 0.
I20260924 04:08:26.271225 1945 raft_consensus.cc:2454] T 7b9b6c8f72b44244883984e3b3d67754 P 0872e2c0377340098c419c30801f9b55 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1d025e13aef436f94ce8c10ca541edf in term 0.
I20260924 04:08:26.271538 1737 leader_election.cc:302] T 7b9b6c8f72b44244883984e3b3d67754 P c1d025e13aef436f94ce8c10ca541edf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7db0dad5cd924a5f80a51795d2a0315d, c1d025e13aef436f94ce8c10ca541edf; no voters:
I20260924 04:08:26.271735 2035 raft_consensus.cc:2781] T 7b9b6c8f72b44244883984e3b3d67754 P c1d025e13aef436f94ce8c10ca541edf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:08:26.271798 2035 raft_consensus.cc:482] T 7b9b6c8f72b44244883984e3b3d67754 P c1d025e13aef436f94ce8c10ca541edf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:08:26.271870 2035 raft_consensus.cc:3037] T 7b9b6c8f72b44244883984e3b3d67754 P c1d025e13aef436f94ce8c10ca541edf [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:08:26.273247 2035 raft_consensus.cc:504] T 7b9b6c8f72b44244883984e3b3d67754 P c1d025e13aef436f94ce8c10ca541edf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db0dad5cd924a5f80a51795d2a0315d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41777 } } peers { permanent_uuid: "c1d025e13aef436f94ce8c10ca541edf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39347 } } peers { permanent_uuid: "0872e2c0377340098c419c30801f9b55" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40375 } }
I20260924 04:08:26.273634 2035 leader_election.cc:288] T 7b9b6c8f72b44244883984e3b3d67754 P c1d025e13aef436f94ce8c10ca541edf [CANDIDATE]: Term 1 election: Requested vote from peers 7db0dad5cd924a5f80a51795d2a0315d (127.0.98.1:41777), 0872e2c0377340098c419c30801f9b55 (127.0.98.3:40375)
I20260924 04:08:26.274278 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b9b6c8f72b44244883984e3b3d67754" candidate_uuid: "c1d025e13aef436f94ce8c10ca541edf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7db0dad5cd924a5f80a51795d2a0315d"
I20260924 04:08:26.274396 1663 raft_consensus.cc:3037] T 7b9b6c8f72b44244883984e3b3d67754 P 7db0dad5cd924a5f80a51795d2a0315d [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:08:26.274405 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b9b6c8f72b44244883984e3b3d67754" candidate_uuid: "c1d025e13aef436f94ce8c10ca541edf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0872e2c0377340098c419c30801f9b55"
I20260924 04:08:26.274506 1945 raft_consensus.cc:3037] T 7b9b6c8f72b44244883984e3b3d67754 P 0872e2c0377340098c419c30801f9b55 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:08:26.275365 1945 raft_consensus.cc:2454] T 7b9b6c8f72b44244883984e3b3d67754 P 0872e2c0377340098c419c30801f9b55 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1d025e13aef436f94ce8c10ca541edf in term 1.
I20260924 04:08:26.275552 1663 raft_consensus.cc:2454] T 7b9b6c8f72b44244883984e3b3d67754 P 7db0dad5cd924a5f80a51795d2a0315d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1d025e13aef436f94ce8c10ca541edf in term 1.
I20260924 04:08:26.275710 1738 leader_election.cc:302] T 7b9b6c8f72b44244883984e3b3d67754 P c1d025e13aef436f94ce8c10ca541edf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0872e2c0377340098c419c30801f9b55, c1d025e13aef436f94ce8c10ca541edf; no voters:
I20260924 04:08:26.275908 2035 raft_consensus.cc:2781] T 7b9b6c8f72b44244883984e3b3d67754 P c1d025e13aef436f94ce8c10ca541edf [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:08:26.276022 2035 raft_consensus.cc:686] T 7b9b6c8f72b44244883984e3b3d67754 P c1d025e13aef436f94ce8c10ca541edf [term 1 LEADER]: Becoming Leader. State: Replica: c1d025e13aef436f94ce8c10ca541edf, State: Running, Role: LEADER
I20260924 04:08:26.276206 2035 consensus_queue.cc:237] T 7b9b6c8f72b44244883984e3b3d67754 P c1d025e13aef436f94ce8c10ca541edf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db0dad5cd924a5f80a51795d2a0315d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41777 } } peers { permanent_uuid: "c1d025e13aef436f94ce8c10ca541edf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39347 } } peers { permanent_uuid: "0872e2c0377340098c419c30801f9b55" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40375 } }
I20260924 04:08:26.277928 1507 catalog_manager.cc:5495] T 7b9b6c8f72b44244883984e3b3d67754 P c1d025e13aef436f94ce8c10ca541edf reported cstate change: term changed from 0 to 1, leader changed from <none> to c1d025e13aef436f94ce8c10ca541edf (127.0.98.2). New cstate: current_term: 1 leader_uuid: "c1d025e13aef436f94ce8c10ca541edf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db0dad5cd924a5f80a51795d2a0315d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41777 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1d025e13aef436f94ce8c10ca541edf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39347 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0872e2c0377340098c419c30801f9b55" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40375 } health_report { overall_health: UNKNOWN } } }
Sep 24 04:08:26 dist-test-slave-00ld krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222904, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:08:26.334564 2050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:26.326370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34100 (local address 127.0.98.62:40335)
0924 04:08:26.326565 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:26.326568 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:26.326687 (+ 119us) server_negotiation.cc:407] Connection header received
0924 04:08:26.326790 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:26.326792 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:26.326832 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:26.326919 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:26.327553 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:26.328094 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:26.328907 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:26.329153 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:26.329233 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:26.331424 (+ 2191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:26.331428 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:26.331430 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:26.331450 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:26.333066 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:26.333469 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:26.333471 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:26.333472 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:26.333545 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:26.333774 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:26.333776 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:26.333777 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:26.333935 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:26.334033 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:26.334295 (+ 262us) server_negotiation.cc:299] Negotiation successful
0924 04:08:26.334386 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":57,"threads_started":1}
I20260924 04:08:26.334597 2051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:26.326332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40335 (local address 127.0.0.1:34100)
0924 04:08:26.326589 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:26.326614 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:26.326712 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:26.326961 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:26.326963 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:26.327351 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:08:26.327438 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:26.327444 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:26.328188 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:26.328191 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:26.328801 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:26.328809 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:26.329252 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:26.329254 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:26.329311 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:26.329851 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:08:26.329862 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:08:26.331300 (+ 1438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:08:26.333159 (+ 1859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:26.333166 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:26.333171 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:26.333385 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:26.333623 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:26.333626 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:26.333627 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:26.333701 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:26.334051 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:08:26.334056 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:08:26.334149 (+ 93us) client_negotiation.cc:769] Sending connection context
0924 04:08:26.334235 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 04:08:26.334337 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":75,"threads_started":1}
I20260924 04:08:26.338716 1507 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34100:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 04:08:26.339483 1507 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:08:26.344326 1645 tablet_service.cc:1467] Processing CreateTablet for tablet 09a435fb520c4e82956ca365d3cad70d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d2c9fe10d48a4185a7964c17d542aadf]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 04:08:26.344326 1785 tablet_service.cc:1467] Processing CreateTablet for tablet 09a435fb520c4e82956ca365d3cad70d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d2c9fe10d48a4185a7964c17d542aadf]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 04:08:26.344506 1925 tablet_service.cc:1467] Processing CreateTablet for tablet 09a435fb520c4e82956ca365d3cad70d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d2c9fe10d48a4185a7964c17d542aadf]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 04:08:26.344769 1785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09a435fb520c4e82956ca365d3cad70d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:08:26.344770 1645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09a435fb520c4e82956ca365d3cad70d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:08:26.344858 1925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09a435fb520c4e82956ca365d3cad70d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:08:26.347235 2016 tablet_bootstrap.cc:492] T 09a435fb520c4e82956ca365d3cad70d P 7db0dad5cd924a5f80a51795d2a0315d: Bootstrap starting.
I20260924 04:08:26.347397 2014 tablet_bootstrap.cc:492] T 09a435fb520c4e82956ca365d3cad70d P c1d025e13aef436f94ce8c10ca541edf: Bootstrap starting.
I20260924 04:08:26.348073 2016 tablet_bootstrap.cc:654] T 09a435fb520c4e82956ca365d3cad70d P 7db0dad5cd924a5f80a51795d2a0315d: Neither blocks nor log segments found. Creating new log.
I20260924 04:08:26.348220 2014 tablet_bootstrap.cc:654] T 09a435fb520c4e82956ca365d3cad70d P c1d025e13aef436f94ce8c10ca541edf: Neither blocks nor log segments found. Creating new log.
I20260924 04:08:26.349062 2015 tablet_bootstrap.cc:492] T 09a435fb520c4e82956ca365d3cad70d P 0872e2c0377340098c419c30801f9b55: Bootstrap starting.
I20260924 04:08:26.349336 2016 tablet_bootstrap.cc:492] T 09a435fb520c4e82956ca365d3cad70d P 7db0dad5cd924a5f80a51795d2a0315d: No bootstrap required, opened a new log
I20260924 04:08:26.349395 2014 tablet_bootstrap.cc:492] T 09a435fb520c4e82956ca365d3cad70d P c1d025e13aef436f94ce8c10ca541edf: No bootstrap required, opened a new log
I20260924 04:08:26.349408 2016 ts_tablet_manager.cc:1397] T 09a435fb520c4e82956ca365d3cad70d P 7db0dad5cd924a5f80a51795d2a0315d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 04:08:26.349459 2014 ts_tablet_manager.cc:1397] T 09a435fb520c4e82956ca365d3cad70d P c1d025e13aef436f94ce8c10ca541edf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 04:08:26.349825 2014 raft_consensus.cc:348] T 09a435fb520c4e82956ca365d3cad70d P c1d025e13aef436f94ce8c10ca541edf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db0dad5cd924a5f80a51795d2a0315d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41777 } } peers { permanent_uuid: "c1d025e13aef436f94ce8c10ca541edf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39347 } } peers { permanent_uuid: "0872e2c0377340098c419c30801f9b55" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40375 } }
I20260924 04:08:26.349825 2016 raft_consensus.cc:348] T 09a435fb520c4e82956ca365d3cad70d P 7db0dad5cd924a5f80a51795d2a0315d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db0dad5cd924a5f80a51795d2a0315d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41777 } } peers { permanent_uuid: "c1d025e13aef436f94ce8c10ca541edf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39347 } } peers { permanent_uuid: "0872e2c0377340098c419c30801f9b55" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40375 } }
I20260924 04:08:26.349929 2016 raft_consensus.cc:374] T 09a435fb520c4e82956ca365d3cad70d P 7db0dad5cd924a5f80a51795d2a0315d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:08:26.349929 2014 raft_consensus.cc:374] T 09a435fb520c4e82956ca365d3cad70d P c1d025e13aef436f94ce8c10ca541edf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:08:26.349963 2016 raft_consensus.cc:729] T 09a435fb520c4e82956ca365d3cad70d P 7db0dad5cd924a5f80a51795d2a0315d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7db0dad5cd924a5f80a51795d2a0315d, State: Initialized, Role: FOLLOWER
I20260924 04:08:26.349963 2014 raft_consensus.cc:729] T 09a435fb520c4e82956ca365d3cad70d P c1d025e13aef436f94ce8c10ca541edf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1d025e13aef436f94ce8c10ca541edf, State: Initialized, Role: FOLLOWER
I20260924 04:08:26.350085 2014 consensus_queue.cc:260] T 09a435fb520c4e82956ca365d3cad70d P c1d025e13aef436f94ce8c10ca541edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db0dad5cd924a5f80a51795d2a0315d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41777 } } peers { permanent_uuid: "c1d025e13aef436f94ce8c10ca541edf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39347 } } peers { permanent_uuid: "0872e2c0377340098c419c30801f9b55" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40375 } }
I20260924 04:08:26.350085 2016 consensus_queue.cc:260] T 09a435fb520c4e82956ca365d3cad70d P 7db0dad5cd924a5f80a51795d2a0315d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db0dad5cd924a5f80a51795d2a0315d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41777 } } peers { permanent_uuid: "c1d025e13aef436f94ce8c10ca541edf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39347 } } peers { permanent_uuid: "0872e2c0377340098c419c30801f9b55" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40375 } }
I20260924 04:08:26.350338 2014 ts_tablet_manager.cc:1428] T 09a435fb520c4e82956ca365d3cad70d P c1d025e13aef436f94ce8c10ca541edf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:08:26.350338 2016 ts_tablet_manager.cc:1428] T 09a435fb520c4e82956ca365d3cad70d P 7db0dad5cd924a5f80a51795d2a0315d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:08:26.350407 2015 tablet_bootstrap.cc:654] T 09a435fb520c4e82956ca365d3cad70d P 0872e2c0377340098c419c30801f9b55: Neither blocks nor log segments found. Creating new log.
I20260924 04:08:26.350453 2035 tablet_replica.cc:439] T 09a435fb520c4e82956ca365d3cad70d P c1d025e13aef436f94ce8c10ca541edf: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db0dad5cd924a5f80a51795d2a0315d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41777 } } peers { permanent_uuid: "c1d025e13aef436f94ce8c10ca541edf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39347 } } peers { permanent_uuid: "0872e2c0377340098c419c30801f9b55" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40375 } } }
I20260924 04:08:26.350586 2035 tablet_replica.cc:442] T 09a435fb520c4e82956ca365d3cad70d P c1d025e13aef436f94ce8c10ca541edf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:08:26.350344 2022 tablet_replica.cc:439] T 09a435fb520c4e82956ca365d3cad70d P 7db0dad5cd924a5f80a51795d2a0315d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db0dad5cd924a5f80a51795d2a0315d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41777 } } peers { permanent_uuid: "c1d025e13aef436f94ce8c10ca541edf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39347 } } peers { permanent_uuid: "0872e2c0377340098c419c30801f9b55" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40375 } } }
I20260924 04:08:26.350963 2022 tablet_replica.cc:442] T 09a435fb520c4e82956ca365d3cad70d P 7db0dad5cd924a5f80a51795d2a0315d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:08:26.351899 2015 tablet_bootstrap.cc:492] T 09a435fb520c4e82956ca365d3cad70d P 0872e2c0377340098c419c30801f9b55: No bootstrap required, opened a new log
I20260924 04:08:26.351969 2015 ts_tablet_manager.cc:1397] T 09a435fb520c4e82956ca365d3cad70d P 0872e2c0377340098c419c30801f9b55: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 04:08:26.352442 2015 raft_consensus.cc:348] T 09a435fb520c4e82956ca365d3cad70d P 0872e2c0377340098c419c30801f9b55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db0dad5cd924a5f80a51795d2a0315d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41777 } } peers { permanent_uuid: "c1d025e13aef436f94ce8c10ca541edf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39347 } } peers { permanent_uuid: "0872e2c0377340098c419c30801f9b55" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40375 } }
I20260924 04:08:26.352533 2015 raft_consensus.cc:374] T 09a435fb520c4e82956ca365d3cad70d P 0872e2c0377340098c419c30801f9b55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:08:26.352583 2015 raft_consensus.cc:729] T 09a435fb520c4e82956ca365d3cad70d P 0872e2c0377340098c419c30801f9b55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0872e2c0377340098c419c30801f9b55, State: Initialized, Role: FOLLOWER
I20260924 04:08:26.352723 2015 consensus_queue.cc:260] T 09a435fb520c4e82956ca365d3cad70d P 0872e2c0377340098c419c30801f9b55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db0dad5cd924a5f80a51795d2a0315d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41777 } } peers { permanent_uuid: "c1d025e13aef436f94ce8c10ca541edf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39347 } } peers { permanent_uuid: "0872e2c0377340098c419c30801f9b55" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40375 } }
I20260924 04:08:26.352993 2015 ts_tablet_manager.cc:1428] T 09a435fb520c4e82956ca365d3cad70d P 0872e2c0377340098c419c30801f9b55: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:08:26.352941 2021 tablet_replica.cc:439] T 09a435fb520c4e82956ca365d3cad70d P 0872e2c0377340098c419c30801f9b55: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db0dad5cd924a5f80a51795d2a0315d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41777 } } peers { permanent_uuid: "c1d025e13aef436f94ce8c10ca541edf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39347 } } peers { permanent_uuid: "0872e2c0377340098c419c30801f9b55" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40375 } } }
I20260924 04:08:26.353106 2021 tablet_replica.cc:442] T 09a435fb520c4e82956ca365d3cad70d P 0872e2c0377340098c419c30801f9b55: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:08:26.545189 2022 raft_consensus.cc:482] T 09a435fb520c4e82956ca365d3cad70d P 7db0dad5cd924a5f80a51795d2a0315d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:08:26.545323 2022 raft_consensus.cc:504] T 09a435fb520c4e82956ca365d3cad70d P 7db0dad5cd924a5f80a51795d2a0315d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db0dad5cd924a5f80a51795d2a0315d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41777 } } peers { permanent_uuid: "c1d025e13aef436f94ce8c10ca541edf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39347 } } peers { permanent_uuid: "0872e2c0377340098c419c30801f9b55" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40375 } }
I20260924 04:08:26.546478 2022 leader_election.cc:288] T 09a435fb520c4e82956ca365d3cad70d P 7db0dad5cd924a5f80a51795d2a0315d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1d025e13aef436f94ce8c10ca541edf (127.0.98.2:39347), 0872e2c0377340098c419c30801f9b55 (127.0.98.3:40375)
I20260924 04:08:26.550917 2056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:26.546739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40375 (local address 127.0.98.1:33325)
0924 04:08:26.547117 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:26.547130 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:26.547228 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:26.547451 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:26.547454 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:26.547467 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:08:26.547585 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:26.547590 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:26.548352 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:26.548355 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:26.549597 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:26.549607 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:26.550434 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:26.550437 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:26.550555 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:26.550609 (+ 54us) client_negotiation.cc:769] Sending connection context
0924 04:08:26.550686 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 04:08:26.550756 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":99,"threads_started":1}
I20260924 04:08:26.550932 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:26.547109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33325 (local address 127.0.98.3:40375)
0924 04:08:26.547223 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:26.547228 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:26.547243 (+ 15us) server_negotiation.cc:407] Connection header received
0924 04:08:26.547302 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:26.547305 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:26.547348 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:26.547425 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:08:26.547697 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:26.548241 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:26.549723 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:26.550320 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:26.550426 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:26.550498 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:26.550727 (+ 229us) server_negotiation.cc:299] Negotiation successful
0924 04:08:26.550785 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 04:08:26.551033 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:26.546752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49411 (local address 127.0.98.2:39347)
0924 04:08:26.546872 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:26.546876 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:26.547132 (+ 256us) server_negotiation.cc:407] Connection header received
0924 04:08:26.547327 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:26.547330 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:26.547379 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:26.547429 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:08:26.547838 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:26.548579 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:26.549894 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:26.550509 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:26.550602 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:26.550646 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:26.550849 (+ 203us) server_negotiation.cc:299] Negotiation successful
0924 04:08:26.550920 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 04:08:26.551049 2055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:26.546657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39347 (local address 127.0.98.1:49411)
0924 04:08:26.547045 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:26.547059 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:26.547207 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:26.547576 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:26.547578 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:26.547592 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:08:26.547727 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:26.547734 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:26.548739 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:26.548742 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:26.549787 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:26.549793 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:26.550632 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:26.550639 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:26.550735 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:26.550753 (+ 18us) client_negotiation.cc:769] Sending connection context
0924 04:08:26.550819 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 04:08:26.550870 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":128,"threads_started":1}
I20260924 04:08:26.551510 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09a435fb520c4e82956ca365d3cad70d" candidate_uuid: "7db0dad5cd924a5f80a51795d2a0315d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0872e2c0377340098c419c30801f9b55" is_pre_election: true
I20260924 04:08:26.551525 1805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09a435fb520c4e82956ca365d3cad70d" candidate_uuid: "7db0dad5cd924a5f80a51795d2a0315d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1d025e13aef436f94ce8c10ca541edf" is_pre_election: true
I20260924 04:08:26.551685 1945 raft_consensus.cc:2454] T 09a435fb520c4e82956ca365d3cad70d P 0872e2c0377340098c419c30801f9b55 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7db0dad5cd924a5f80a51795d2a0315d in term 0.
I20260924 04:08:26.551748 1805 raft_consensus.cc:2454] T 09a435fb520c4e82956ca365d3cad70d P c1d025e13aef436f94ce8c10ca541edf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7db0dad5cd924a5f80a51795d2a0315d in term 0.
I20260924 04:08:26.552078 1598 leader_election.cc:302] T 09a435fb520c4e82956ca365d3cad70d P 7db0dad5cd924a5f80a51795d2a0315d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0872e2c0377340098c419c30801f9b55, 7db0dad5cd924a5f80a51795d2a0315d; no voters:
I20260924 04:08:26.552388 2022 raft_consensus.cc:2781] T 09a435fb520c4e82956ca365d3cad70d P 7db0dad5cd924a5f80a51795d2a0315d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:08:26.552451 2022 raft_consensus.cc:482] T 09a435fb520c4e82956ca365d3cad70d P 7db0dad5cd924a5f80a51795d2a0315d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:08:26.552520 2022 raft_consensus.cc:3037] T 09a435fb520c4e82956ca365d3cad70d P 7db0dad5cd924a5f80a51795d2a0315d [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:08:26.553494 2022 raft_consensus.cc:504] T 09a435fb520c4e82956ca365d3cad70d P 7db0dad5cd924a5f80a51795d2a0315d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db0dad5cd924a5f80a51795d2a0315d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41777 } } peers { permanent_uuid: "c1d025e13aef436f94ce8c10ca541edf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39347 } } peers { permanent_uuid: "0872e2c0377340098c419c30801f9b55" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40375 } }
I20260924 04:08:26.553879 2022 leader_election.cc:288] T 09a435fb520c4e82956ca365d3cad70d P 7db0dad5cd924a5f80a51795d2a0315d [CANDIDATE]: Term 1 election: Requested vote from peers c1d025e13aef436f94ce8c10ca541edf (127.0.98.2:39347), 0872e2c0377340098c419c30801f9b55 (127.0.98.3:40375)
I20260924 04:08:26.554211 1805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09a435fb520c4e82956ca365d3cad70d" candidate_uuid: "7db0dad5cd924a5f80a51795d2a0315d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1d025e13aef436f94ce8c10ca541edf"
I20260924 04:08:26.554270 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09a435fb520c4e82956ca365d3cad70d" candidate_uuid: "7db0dad5cd924a5f80a51795d2a0315d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0872e2c0377340098c419c30801f9b55"
I20260924 04:08:26.554342 1805 raft_consensus.cc:3037] T 09a435fb520c4e82956ca365d3cad70d P c1d025e13aef436f94ce8c10ca541edf [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:08:26.554378 1945 raft_consensus.cc:3037] T 09a435fb520c4e82956ca365d3cad70d P 0872e2c0377340098c419c30801f9b55 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:08:26.555505 1945 raft_consensus.cc:2454] T 09a435fb520c4e82956ca365d3cad70d P 0872e2c0377340098c419c30801f9b55 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7db0dad5cd924a5f80a51795d2a0315d in term 1.
I20260924 04:08:26.555541 1805 raft_consensus.cc:2454] T 09a435fb520c4e82956ca365d3cad70d P c1d025e13aef436f94ce8c10ca541edf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7db0dad5cd924a5f80a51795d2a0315d in term 1.
I20260924 04:08:26.555812 1598 leader_election.cc:302] T 09a435fb520c4e82956ca365d3cad70d P 7db0dad5cd924a5f80a51795d2a0315d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0872e2c0377340098c419c30801f9b55, 7db0dad5cd924a5f80a51795d2a0315d; no voters:
I20260924 04:08:26.556094 2022 raft_consensus.cc:2781] T 09a435fb520c4e82956ca365d3cad70d P 7db0dad5cd924a5f80a51795d2a0315d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:08:26.556282 2022 raft_consensus.cc:686] T 09a435fb520c4e82956ca365d3cad70d P 7db0dad5cd924a5f80a51795d2a0315d [term 1 LEADER]: Becoming Leader. State: Replica: 7db0dad5cd924a5f80a51795d2a0315d, State: Running, Role: LEADER
I20260924 04:08:26.556551 2022 consensus_queue.cc:237] T 09a435fb520c4e82956ca365d3cad70d P 7db0dad5cd924a5f80a51795d2a0315d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db0dad5cd924a5f80a51795d2a0315d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41777 } } peers { permanent_uuid: "c1d025e13aef436f94ce8c10ca541edf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39347 } } peers { permanent_uuid: "0872e2c0377340098c419c30801f9b55" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40375 } }
I20260924 04:08:26.557957 2057 tablet_replica.cc:439] T 09a435fb520c4e82956ca365d3cad70d P 7db0dad5cd924a5f80a51795d2a0315d: TxnStatusTablet state changed. Reason: New leader 7db0dad5cd924a5f80a51795d2a0315d. Latest consensus state: current_term: 1 leader_uuid: "7db0dad5cd924a5f80a51795d2a0315d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db0dad5cd924a5f80a51795d2a0315d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41777 } } peers { permanent_uuid: "c1d025e13aef436f94ce8c10ca541edf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39347 } } peers { permanent_uuid: "0872e2c0377340098c419c30801f9b55" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40375 } } }
I20260924 04:08:26.558068 2057 tablet_replica.cc:442] T 09a435fb520c4e82956ca365d3cad70d P 7db0dad5cd924a5f80a51795d2a0315d: This TxnStatusTablet replica's current role is: LEADER
I20260924 04:08:26.558348 2059 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 04:08:26.558525 2059 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 04:08:26.559412 1507 catalog_manager.cc:5495] T 09a435fb520c4e82956ca365d3cad70d P 7db0dad5cd924a5f80a51795d2a0315d reported cstate change: term changed from 0 to 1, leader changed from <none> to 7db0dad5cd924a5f80a51795d2a0315d (127.0.98.1). New cstate: current_term: 1 leader_uuid: "7db0dad5cd924a5f80a51795d2a0315d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db0dad5cd924a5f80a51795d2a0315d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41777 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c1d025e13aef436f94ce8c10ca541edf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0872e2c0377340098c419c30801f9b55" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40375 } health_report { overall_health: UNKNOWN } } }
I20260924 04:08:26.589547 2051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:26.585670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41777 (local address 127.0.0.1:53840)
0924 04:08:26.585812 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:26.585830 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:26.585927 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:26.586383 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:26.586387 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:26.586396 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 04:08:26.586504 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:26.586509 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:26.587274 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:26.587280 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:26.587886 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:26.587916 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:26.588369 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:26.588373 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:26.588471 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:26.588510 (+ 39us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 04:08:26.589249 (+ 739us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 04:08:26.589263 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 04:08:26.589325 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 04:08:26.589410 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260924 04:08:26.589572 2062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:26.585803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53840 (local address 127.0.98.1:41777)
0924 04:08:26.586107 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:26.586115 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:26.586126 (+ 11us) server_negotiation.cc:407] Connection header received
0924 04:08:26.586197 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:26.586199 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:26.586253 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:26.586373 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:08:26.586636 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:26.587166 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:26.588028 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:26.588260 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:26.588370 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:26.588718 (+ 348us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:08:26.589154 (+ 436us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:08:26.589277 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:26.589347 (+ 70us) server_negotiation.cc:299] Negotiation successful
0924 04:08:26.589395 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":111,"threads_started":1}
I20260924 04:08:26.592536 1945 raft_consensus.cc:1261] T 09a435fb520c4e82956ca365d3cad70d P 0872e2c0377340098c419c30801f9b55 [term 1 FOLLOWER]: Refusing update from remote peer 7db0dad5cd924a5f80a51795d2a0315d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:08:26.592653 1805 raft_consensus.cc:1261] T 09a435fb520c4e82956ca365d3cad70d P c1d025e13aef436f94ce8c10ca541edf [term 1 FOLLOWER]: Refusing update from remote peer 7db0dad5cd924a5f80a51795d2a0315d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:08:26.593145 2057 consensus_queue.cc:1035] T 09a435fb520c4e82956ca365d3cad70d P 7db0dad5cd924a5f80a51795d2a0315d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0872e2c0377340098c419c30801f9b55" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:08:26.593349 2022 consensus_queue.cc:1035] T 09a435fb520c4e82956ca365d3cad70d P 7db0dad5cd924a5f80a51795d2a0315d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1d025e13aef436f94ce8c10ca541edf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:08:26.595021 2021 tablet_replica.cc:439] T 09a435fb520c4e82956ca365d3cad70d P 0872e2c0377340098c419c30801f9b55: TxnStatusTablet state changed. Reason: New leader 7db0dad5cd924a5f80a51795d2a0315d. Latest consensus state: current_term: 1 leader_uuid: "7db0dad5cd924a5f80a51795d2a0315d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db0dad5cd924a5f80a51795d2a0315d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41777 } } peers { permanent_uuid: "c1d025e13aef436f94ce8c10ca541edf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39347 } } peers { permanent_uuid: "0872e2c0377340098c419c30801f9b55" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40375 } } }
I20260924 04:08:26.595014 2035 tablet_replica.cc:439] T 09a435fb520c4e82956ca365d3cad70d P c1d025e13aef436f94ce8c10ca541edf: TxnStatusTablet state changed. Reason: New leader 7db0dad5cd924a5f80a51795d2a0315d. Latest consensus state: current_term: 1 leader_uuid: "7db0dad5cd924a5f80a51795d2a0315d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db0dad5cd924a5f80a51795d2a0315d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41777 } } peers { permanent_uuid: "c1d025e13aef436f94ce8c10ca541edf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39347 } } peers { permanent_uuid: "0872e2c0377340098c419c30801f9b55" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40375 } } }
I20260924 04:08:26.595132 2021 tablet_replica.cc:442] T 09a435fb520c4e82956ca365d3cad70d P 0872e2c0377340098c419c30801f9b55: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:08:26.595132 2035 tablet_replica.cc:442] T 09a435fb520c4e82956ca365d3cad70d P c1d025e13aef436f94ce8c10ca541edf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:08:26.597218 2022 tablet_replica.cc:439] T 09a435fb520c4e82956ca365d3cad70d P 7db0dad5cd924a5f80a51795d2a0315d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7db0dad5cd924a5f80a51795d2a0315d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db0dad5cd924a5f80a51795d2a0315d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41777 } } peers { permanent_uuid: "c1d025e13aef436f94ce8c10ca541edf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39347 } } peers { permanent_uuid: "0872e2c0377340098c419c30801f9b55" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40375 } } }
I20260924 04:08:26.597344 2022 tablet_replica.cc:442] T 09a435fb520c4e82956ca365d3cad70d P 7db0dad5cd924a5f80a51795d2a0315d: This TxnStatusTablet replica's current role is: LEADER
I20260924 04:08:26.597441 2030 mvcc.cc:205] Tried to move back new op lower bound from 7332753025398013952 to 7332753025258287104. Current Snapshot: MvccSnapshot[applied={T|T < 7332753025398013952 or (T in {7332753025398013952})}]
I20260924 04:08:26.597852 2029 mvcc.cc:205] Tried to move back new op lower bound from 7332753025398013952 to 7332753025258287104. Current Snapshot: MvccSnapshot[applied={T|T < 7332753025398013952}]
I20260924 04:08:26.597926 2022 tablet_replica.cc:439] T 09a435fb520c4e82956ca365d3cad70d P 7db0dad5cd924a5f80a51795d2a0315d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7db0dad5cd924a5f80a51795d2a0315d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db0dad5cd924a5f80a51795d2a0315d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41777 } } peers { permanent_uuid: "c1d025e13aef436f94ce8c10ca541edf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39347 } } peers { permanent_uuid: "0872e2c0377340098c419c30801f9b55" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40375 } } }
I20260924 04:08:26.598018 2022 tablet_replica.cc:442] T 09a435fb520c4e82956ca365d3cad70d P 7db0dad5cd924a5f80a51795d2a0315d: This TxnStatusTablet replica's current role is: LEADER
I20260924 04:08:26.598014 2021 tablet_replica.cc:439] T 09a435fb520c4e82956ca365d3cad70d P 0872e2c0377340098c419c30801f9b55: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7db0dad5cd924a5f80a51795d2a0315d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db0dad5cd924a5f80a51795d2a0315d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41777 } } peers { permanent_uuid: "c1d025e13aef436f94ce8c10ca541edf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39347 } } peers { permanent_uuid: "0872e2c0377340098c419c30801f9b55" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40375 } } }
I20260924 04:08:26.598100 2021 tablet_replica.cc:442] T 09a435fb520c4e82956ca365d3cad70d P 0872e2c0377340098c419c30801f9b55: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:08:26.603006 2035 tablet_replica.cc:439] T 09a435fb520c4e82956ca365d3cad70d P c1d025e13aef436f94ce8c10ca541edf: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7db0dad5cd924a5f80a51795d2a0315d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db0dad5cd924a5f80a51795d2a0315d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41777 } } peers { permanent_uuid: "c1d025e13aef436f94ce8c10ca541edf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39347 } } peers { permanent_uuid: "0872e2c0377340098c419c30801f9b55" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40375 } } }
I20260924 04:08:26.603129 2035 tablet_replica.cc:442] T 09a435fb520c4e82956ca365d3cad70d P c1d025e13aef436f94ce8c10ca541edf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:08:26.612018 2062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:26.608658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53339 (local address 127.0.98.1:41777)
0924 04:08:26.608792 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:26.608797 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:26.608922 (+ 125us) server_negotiation.cc:407] Connection header received
0924 04:08:26.609043 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:26.609045 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:26.609086 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:26.609147 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:08:26.609388 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:26.609893 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:26.611082 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:26.611278 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:26.611351 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:26.611657 (+ 306us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:08:26.611718 (+ 61us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:08:26.611770 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:26.611878 (+ 108us) server_negotiation.cc:299] Negotiation successful
0924 04:08:26.611918 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 04:08:26.612093 2067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:26.608510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41777 (local address 127.0.98.2:53339)
0924 04:08:26.608838 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:26.608854 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:26.608957 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:26.609177 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:26.609180 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:26.609188 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 04:08:26.609292 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:26.609298 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:26.609986 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:26.609990 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:26.610972 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:26.610982 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:26.611380 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:26.611383 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:26.611476 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:26.611534 (+ 58us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 04:08:26.611794 (+ 260us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 04:08:26.611800 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 04:08:26.611864 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 04:08:26.611949 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":124,"threads_started":1}
I20260924 04:08:26.618525 1663 raft_consensus.cc:1261] T 7b9b6c8f72b44244883984e3b3d67754 P 7db0dad5cd924a5f80a51795d2a0315d [term 1 FOLLOWER]: Refusing update from remote peer c1d025e13aef436f94ce8c10ca541edf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:08:26.618602 1945 raft_consensus.cc:1261] T 7b9b6c8f72b44244883984e3b3d67754 P 0872e2c0377340098c419c30801f9b55 [term 1 FOLLOWER]: Refusing update from remote peer c1d025e13aef436f94ce8c10ca541edf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:08:26.618919 2035 consensus_queue.cc:1035] T 7b9b6c8f72b44244883984e3b3d67754 P c1d025e13aef436f94ce8c10ca541edf [LEADER]: Connected to new peer: Peer: permanent_uuid: "7db0dad5cd924a5f80a51795d2a0315d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:08:26.619045 2025 consensus_queue.cc:1035] T 7b9b6c8f72b44244883984e3b3d67754 P c1d025e13aef436f94ce8c10ca541edf [LEADER]: Connected to new peer: Peer: permanent_uuid: "0872e2c0377340098c419c30801f9b55" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:08:26.641646 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:26.637638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58717 (local address 127.0.98.2:39347)
0924 04:08:26.637744 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:26.637749 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:26.638012 (+ 263us) server_negotiation.cc:407] Connection header received
0924 04:08:26.638166 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:26.638169 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:26.638217 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:26.638301 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:08:26.638530 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:26.639298 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:26.640554 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:26.640848 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:26.640952 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:26.641263 (+ 311us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:08:26.641328 (+ 65us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:08:26.641387 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:26.641489 (+ 102us) server_negotiation.cc:299] Negotiation successful
0924 04:08:26.641533 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 04:08:26.641887 2072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:26.637600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39347 (local address 127.0.98.1:58717)
0924 04:08:26.637930 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:26.637943 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:26.638062 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:26.638306 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:26.638308 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:26.638314 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 04:08:26.638435 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:26.638439 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:26.639424 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:26.639428 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:26.640444 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:26.640454 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:26.640980 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:26.640984 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:26.641082 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:26.641133 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 04:08:26.641409 (+ 276us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 04:08:26.641415 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 04:08:26.641489 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 04:08:26.641565 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":114,"threads_started":1}
I20260924 04:08:26.666404 1508 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34088:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:08:26.666603 1508 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34088:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:08:26.668535 1508 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a9550222f0fa4b488486b4d4ff95b688: Sending DeleteTablet for 3 replicas of tablet 7b9b6c8f72b44244883984e3b3d67754
I20260924 04:08:26.669130 1785 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b9b6c8f72b44244883984e3b3d67754 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:08:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53822
I20260924 04:08:26.669275 1925 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b9b6c8f72b44244883984e3b3d67754 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:08:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39916
I20260924 04:08:26.669486 2075 tablet_replica.cc:330] T 7b9b6c8f72b44244883984e3b3d67754 P c1d025e13aef436f94ce8c10ca541edf: stopping tablet replica
I20260924 04:08:26.669627 2075 raft_consensus.cc:2229] T 7b9b6c8f72b44244883984e3b3d67754 P c1d025e13aef436f94ce8c10ca541edf [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:08:26.669840 2075 raft_consensus.cc:2258] T 7b9b6c8f72b44244883984e3b3d67754 P c1d025e13aef436f94ce8c10ca541edf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:08:26.670287 1645 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b9b6c8f72b44244883984e3b3d67754 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:08:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53836
I20260924 04:08:26.670594 2076 tablet_replica.cc:330] T 7b9b6c8f72b44244883984e3b3d67754 P 0872e2c0377340098c419c30801f9b55: stopping tablet replica
I20260924 04:08:26.670606 2075 ts_tablet_manager.cc:1905] T 7b9b6c8f72b44244883984e3b3d67754 P c1d025e13aef436f94ce8c10ca541edf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:08:26.670717 2076 raft_consensus.cc:2229] T 7b9b6c8f72b44244883984e3b3d67754 P 0872e2c0377340098c419c30801f9b55 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:08:26.670838 2076 raft_consensus.cc:2258] T 7b9b6c8f72b44244883984e3b3d67754 P 0872e2c0377340098c419c30801f9b55 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:08:26.671489 2076 ts_tablet_manager.cc:1905] T 7b9b6c8f72b44244883984e3b3d67754 P 0872e2c0377340098c419c30801f9b55: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:08:26.673740 2075 ts_tablet_manager.cc:1918] T 7b9b6c8f72b44244883984e3b3d67754 P c1d025e13aef436f94ce8c10ca541edf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 04:08:26.673826 2075 log.cc:1196] T 7b9b6c8f72b44244883984e3b3d67754 P c1d025e13aef436f94ce8c10ca541edf: Deleting WAL directory at /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222898807720-392-0/minicluster-data/ts-1/wal/wals/7b9b6c8f72b44244883984e3b3d67754
I20260924 04:08:26.674121 2075 ts_tablet_manager.cc:1939] T 7b9b6c8f72b44244883984e3b3d67754 P c1d025e13aef436f94ce8c10ca541edf: Deleting consensus metadata
I20260924 04:08:26.674259 2076 ts_tablet_manager.cc:1918] T 7b9b6c8f72b44244883984e3b3d67754 P 0872e2c0377340098c419c30801f9b55: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 04:08:26.674345 2076 log.cc:1196] T 7b9b6c8f72b44244883984e3b3d67754 P 0872e2c0377340098c419c30801f9b55: Deleting WAL directory at /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222898807720-392-0/minicluster-data/ts-2/wal/wals/7b9b6c8f72b44244883984e3b3d67754
I20260924 04:08:26.674371 2078 tablet_replica.cc:330] T 7b9b6c8f72b44244883984e3b3d67754 P 7db0dad5cd924a5f80a51795d2a0315d: stopping tablet replica
I20260924 04:08:26.674497 2078 raft_consensus.cc:2229] T 7b9b6c8f72b44244883984e3b3d67754 P 7db0dad5cd924a5f80a51795d2a0315d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:08:26.674605 2076 ts_tablet_manager.cc:1939] T 7b9b6c8f72b44244883984e3b3d67754 P 0872e2c0377340098c419c30801f9b55: Deleting consensus metadata
I20260924 04:08:26.674636 2078 raft_consensus.cc:2258] T 7b9b6c8f72b44244883984e3b3d67754 P 7db0dad5cd924a5f80a51795d2a0315d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:08:26.675382 1495 catalog_manager.cc:4841] TS c1d025e13aef436f94ce8c10ca541edf (127.0.98.2:39347): tablet 7b9b6c8f72b44244883984e3b3d67754 (table test-table [id=2a863ed1a75d4d87b331c07416fce0dc]) successfully deleted
I20260924 04:08:26.675436 1494 catalog_manager.cc:4841] TS 0872e2c0377340098c419c30801f9b55 (127.0.98.3:40375): tablet 7b9b6c8f72b44244883984e3b3d67754 (table test-table [id=2a863ed1a75d4d87b331c07416fce0dc]) successfully deleted
I20260924 04:08:26.676064 2078 ts_tablet_manager.cc:1905] T 7b9b6c8f72b44244883984e3b3d67754 P 7db0dad5cd924a5f80a51795d2a0315d: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 24 04:08:26 dist-test-slave-00ld krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222905, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 04:08:26.679291 2078 ts_tablet_manager.cc:1918] T 7b9b6c8f72b44244883984e3b3d67754 P 7db0dad5cd924a5f80a51795d2a0315d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 04:08:26.679397 2078 log.cc:1196] T 7b9b6c8f72b44244883984e3b3d67754 P 7db0dad5cd924a5f80a51795d2a0315d: Deleting WAL directory at /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790222898807720-392-0/minicluster-data/ts-0/wal/wals/7b9b6c8f72b44244883984e3b3d67754
I20260924 04:08:26.679697 2078 ts_tablet_manager.cc:1939] T 7b9b6c8f72b44244883984e3b3d67754 P 7db0dad5cd924a5f80a51795d2a0315d: Deleting consensus metadata
I20260924 04:08:26.680459 1493 catalog_manager.cc:4841] TS 7db0dad5cd924a5f80a51795d2a0315d (127.0.98.1:41777): tablet 7b9b6c8f72b44244883984e3b3d67754 (table test-table [id=2a863ed1a75d4d87b331c07416fce0dc]) successfully deleted
I20260924 04:08:26.683480 2062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:26.670380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53844 (local address 127.0.98.1:41777)
0924 04:08:26.670485 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:26.670490 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:26.671131 (+ 641us) server_negotiation.cc:407] Connection header received
0924 04:08:26.671318 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:26.671321 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:26.671370 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:26.671419 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:26.672386 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:26.673150 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:26.674887 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:26.675184 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:26.675282 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:26.678835 (+ 3553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:26.678844 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:26.678860 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:26.678896 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:26.681464 (+ 2568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:26.681940 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:26.681947 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:26.681949 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:26.682060 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:26.682340 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:26.682345 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:26.682347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:26.682596 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:26.682815 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:26.682970 (+ 155us) server_negotiation.cc:299] Negotiation successful
0924 04:08:26.683103 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
W20260924 04:08:26.683938 1692 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53844
I20260924 04:08:26.693193 2050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:26.685676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34106 (local address 127.0.98.62:40335)
0924 04:08:26.685814 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:26.685818 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:26.685866 (+ 48us) server_negotiation.cc:407] Connection header received
0924 04:08:26.685994 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:26.685998 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:26.686053 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:26.686142 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:26.686689 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:26.687472 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:26.688271 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:26.688515 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:26.688627 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:26.689934 (+ 1307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:26.689937 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:26.689940 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:26.689968 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:26.691520 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:26.691953 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:26.691956 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:26.691958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:26.692040 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:26.692310 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:26.692315 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:26.692317 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:26.692494 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:26.692660 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:26.692907 (+ 247us) server_negotiation.cc:299] Negotiation successful
0924 04:08:26.693030 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260924 04:08:26.693593 1508 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34106
I20260924 04:08:26.694768 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 1580
I20260924 04:08:26.701166 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 1720
I20260924 04:08:26.707682 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 1860
I20260924 04:08:26.713775 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 1476
2026-09-24T04:08:26Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3048 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:08:26 dist-test-slave-00ld krb5kdc[2084](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:08:26 dist-test-slave-00ld krb5kdc[2084](info): set up 2 sockets
Sep 24 04:08:26 dist-test-slave-00ld krb5kdc[2084](info): commencing operation
krb5kdc: starting...
W20260924 04:08:27.776288 392 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260924 04:08:27.923661 392 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.147s user 0.000s sys 0.002s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260924 04:08:28.038064 392 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.114s user 0.000s sys 0.002s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/krb5kdc/test-user.keytab.
Sep 24 04:08:28 dist-test-slave-00ld krb5kdc[2084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:08:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:08:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:08:28.152351 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35555
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35555
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:08:28.261206 2097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:08:28.261474 2097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:08:28.261515 2097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:08:28.265673 2097 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:08:28.265731 2097 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 04:08:28.265748 2097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:08:28.265766 2097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:08:28.265784 2097 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:08:28.265798 2097 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:08:28.270985 2097 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35555
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35555
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2097
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:08:28.272318 2097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:08:28.273720 2097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:08:28.284302 2102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:08:28.284298 2105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:08:28.284469 2103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:08:28.284660 2097 server_base.cc:1034] running on GCE node
I20260924 04:08:28.285235 2097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:08:28.286453 2097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:08:28.287611 2097 hybrid_clock.cc:648] HybridClock initialized: now 1790222908287588 us; error 55 us; skew 500 ppm
Sep 24 04:08:28 dist-test-slave-00ld krb5kdc[2084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222908, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:08:28.290310 2097 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:08:28.291153 2097 webserver.cc:460] Webserver started at http://127.0.98.62:34691/ using document root <none> and password file <none>
I20260924 04:08:28.291709 2097 fs_manager.cc:359] Metadata directory not provided
I20260924 04:08:28.291803 2097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:08:28.292042 2097 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:08:28.294104 2097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/minicluster-data/master-0/data/instance:
uuid: "e422a7c9985241bf963fe50b6dedba65"
format_stamp: "Formatted at 2026-09-24 04:08:28 on dist-test-slave-00ld"
server_key: "1a72c26615e118e6880aa217adde6758"
server_key_iv: "e002915c8d6caf3721bdb0abebe3244d"
server_key_version: "encryptionkey@0"
I20260924 04:08:28.294654 2097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/minicluster-data/master-0/wal/instance:
uuid: "e422a7c9985241bf963fe50b6dedba65"
format_stamp: "Formatted at 2026-09-24 04:08:28 on dist-test-slave-00ld"
server_key: "1a72c26615e118e6880aa217adde6758"
server_key_iv: "e002915c8d6caf3721bdb0abebe3244d"
server_key_version: "encryptionkey@0"
I20260924 04:08:28.298210 2097 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260924 04:08:28.300511 2112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:28.301751 2097 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:08:28.301904 2097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/minicluster-data/master-0/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/minicluster-data/master-0/wal
uuid: "e422a7c9985241bf963fe50b6dedba65"
format_stamp: "Formatted at 2026-09-24 04:08:28 on dist-test-slave-00ld"
server_key: "1a72c26615e118e6880aa217adde6758"
server_key_iv: "e002915c8d6caf3721bdb0abebe3244d"
server_key_version: "encryptionkey@0"
I20260924 04:08:28.302016 2097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:08:28.363335 2097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:08:28.366552 2097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:08:28.366771 2097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:08:28.374401 2097 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35555
I20260924 04:08:28.374425 2174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35555 every 8 connection(s)
I20260924 04:08:28.375340 2097 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:08:28.378166 2175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:08:28.379197 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 2097
I20260924 04:08:28.379288 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/minicluster-data/master-0/wal/instance
I20260924 04:08:28.379582 392 external_mini_cluster.cc:1428] Setting key 3058e84c3fcb32cca220883d87f44d72
I20260924 04:08:28.384963 2175 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e422a7c9985241bf963fe50b6dedba65: Bootstrap starting.
I20260924 04:08:28.387642 2175 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e422a7c9985241bf963fe50b6dedba65: Neither blocks nor log segments found. Creating new log.
I20260924 04:08:28.388396 2175 log.cc:824] T 00000000000000000000000000000000 P e422a7c9985241bf963fe50b6dedba65: Log is configured to *not* fsync() on all Append() calls
Sep 24 04:08:28 dist-test-slave-00ld krb5kdc[2084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:08:28.390712 2175 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e422a7c9985241bf963fe50b6dedba65: No bootstrap required, opened a new log
I20260924 04:08:28.395047 2178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:28.381062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53406 (local address 127.0.98.62:35555)
0924 04:08:28.381495 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:28.381514 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:28.381561 (+ 47us) server_negotiation.cc:407] Connection header received
0924 04:08:28.382382 (+ 821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:28.382405 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:28.382813 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:28.383180 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:28.383823 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:28.385270 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:28.386157 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:28.386947 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:28.387111 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:28.389837 (+ 2726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:28.389843 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:28.389859 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:28.389892 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:28.392041 (+ 2149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:28.392453 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:28.392459 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:28.392465 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:28.392596 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:28.392900 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:28.392903 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:28.392905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:28.393403 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:28.393587 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:28.393813 (+ 226us) server_negotiation.cc:299] Negotiation successful
0924 04:08:28.394101 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":114,"threads_started":1}
I20260924 04:08:28.395069 2175 raft_consensus.cc:348] T 00000000000000000000000000000000 P e422a7c9985241bf963fe50b6dedba65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e422a7c9985241bf963fe50b6dedba65" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35555 } }
I20260924 04:08:28.395402 2175 raft_consensus.cc:374] T 00000000000000000000000000000000 P e422a7c9985241bf963fe50b6dedba65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:08:28.395485 2175 raft_consensus.cc:729] T 00000000000000000000000000000000 P e422a7c9985241bf963fe50b6dedba65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e422a7c9985241bf963fe50b6dedba65, State: Initialized, Role: FOLLOWER
I20260924 04:08:28.395891 2175 consensus_queue.cc:260] T 00000000000000000000000000000000 P e422a7c9985241bf963fe50b6dedba65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e422a7c9985241bf963fe50b6dedba65" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35555 } }
I20260924 04:08:28.396040 2175 raft_consensus.cc:388] T 00000000000000000000000000000000 P e422a7c9985241bf963fe50b6dedba65 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:08:28.396119 2175 raft_consensus.cc:482] T 00000000000000000000000000000000 P e422a7c9985241bf963fe50b6dedba65 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:08:28.396211 2175 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e422a7c9985241bf963fe50b6dedba65 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:08:28.397198 2175 raft_consensus.cc:504] T 00000000000000000000000000000000 P e422a7c9985241bf963fe50b6dedba65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e422a7c9985241bf963fe50b6dedba65" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35555 } }
I20260924 04:08:28.397567 2175 leader_election.cc:302] T 00000000000000000000000000000000 P e422a7c9985241bf963fe50b6dedba65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e422a7c9985241bf963fe50b6dedba65; no voters:
I20260924 04:08:28.397823 2175 leader_election.cc:288] T 00000000000000000000000000000000 P e422a7c9985241bf963fe50b6dedba65 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:08:28.397985 2180 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e422a7c9985241bf963fe50b6dedba65 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:08:28.398236 2180 raft_consensus.cc:686] T 00000000000000000000000000000000 P e422a7c9985241bf963fe50b6dedba65 [term 1 LEADER]: Becoming Leader. State: Replica: e422a7c9985241bf963fe50b6dedba65, State: Running, Role: LEADER
I20260924 04:08:28.398591 2180 consensus_queue.cc:237] T 00000000000000000000000000000000 P e422a7c9985241bf963fe50b6dedba65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e422a7c9985241bf963fe50b6dedba65" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35555 } }
I20260924 04:08:28.399080 2175 sys_catalog.cc:564] T 00000000000000000000000000000000 P e422a7c9985241bf963fe50b6dedba65 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:08:28.400446 2182 sys_catalog.cc:455] T 00000000000000000000000000000000 P e422a7c9985241bf963fe50b6dedba65 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e422a7c9985241bf963fe50b6dedba65. Latest consensus state: current_term: 1 leader_uuid: "e422a7c9985241bf963fe50b6dedba65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e422a7c9985241bf963fe50b6dedba65" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35555 } } }
I20260924 04:08:28.400446 2181 sys_catalog.cc:455] T 00000000000000000000000000000000 P e422a7c9985241bf963fe50b6dedba65 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e422a7c9985241bf963fe50b6dedba65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e422a7c9985241bf963fe50b6dedba65" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35555 } } }
I20260924 04:08:28.400612 2181 sys_catalog.cc:458] T 00000000000000000000000000000000 P e422a7c9985241bf963fe50b6dedba65 [sys.catalog]: This master's current role is: LEADER
I20260924 04:08:28.400612 2182 sys_catalog.cc:458] T 00000000000000000000000000000000 P e422a7c9985241bf963fe50b6dedba65 [sys.catalog]: This master's current role is: LEADER
W20260924 04:08:28.403841 2195 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e422a7c9985241bf963fe50b6dedba65: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:08:28.403927 2195 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:08:28.403995 2194 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:08:28.404776 2194 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:08:28.410280 2194 catalog_manager.cc:1347] Generated new cluster ID: ad01e63f3f4f40a18039ba80c6df1a1f
I20260924 04:08:28.410343 2194 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:08:28.434333 2194 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:08:28.435640 2194 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:08:28.451725 2194 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e422a7c9985241bf963fe50b6dedba65: Generated new TSK 0
I20260924 04:08:28.452661 2194 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:08:28.484825 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35555
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:08:28.593650 2201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:08:28.593892 2201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:08:28.593938 2201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:08:28.597961 2201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:08:28.598011 2201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:08:28.598073 2201 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 04:08:28.598102 2201 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 04:08:28.598160 2201 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:08:28.603583 2201 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:35555
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2201
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:08:28.604557 2201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:08:28.605597 2201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:08:28.615109 2206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:08:28.615154 2209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:08:28.615155 2201 server_base.cc:1034] running on GCE node
W20260924 04:08:28.615113 2207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:08:28.615747 2201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:08:28.616349 2201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:08:28.617528 2201 hybrid_clock.cc:648] HybridClock initialized: now 1790222908617514 us; error 27 us; skew 500 ppm
Sep 24 04:08:28 dist-test-slave-00ld krb5kdc[2084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222908, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:08:28.620196 2201 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:08:28.621171 2201 webserver.cc:460] Webserver started at http://127.0.98.1:41703/ using document root <none> and password file <none>
I20260924 04:08:28.621721 2201 fs_manager.cc:359] Metadata directory not provided
I20260924 04:08:28.621809 2201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:08:28.622036 2201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:08:28.623994 2201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/minicluster-data/ts-0/data/instance:
uuid: "8c9d6c009d9642109c86c0cd26db2f50"
format_stamp: "Formatted at 2026-09-24 04:08:28 on dist-test-slave-00ld"
server_key: "d82129b54437a254c614123b1db2cfd7"
server_key_iv: "b2988767208d93d5d4f2bc85c7787105"
server_key_version: "encryptionkey@0"
I20260924 04:08:28.624563 2201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8c9d6c009d9642109c86c0cd26db2f50"
format_stamp: "Formatted at 2026-09-24 04:08:28 on dist-test-slave-00ld"
server_key: "d82129b54437a254c614123b1db2cfd7"
server_key_iv: "b2988767208d93d5d4f2bc85c7787105"
server_key_version: "encryptionkey@0"
I20260924 04:08:28.628027 2201 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260924 04:08:28.630354 2216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:28.631487 2201 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:08:28.631618 2201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/minicluster-data/ts-0/wal
uuid: "8c9d6c009d9642109c86c0cd26db2f50"
format_stamp: "Formatted at 2026-09-24 04:08:28 on dist-test-slave-00ld"
server_key: "d82129b54437a254c614123b1db2cfd7"
server_key_iv: "b2988767208d93d5d4f2bc85c7787105"
server_key_version: "encryptionkey@0"
I20260924 04:08:28.631721 2201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:08:28.657501 2201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:08:28.660990 2201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:08:28.661194 2201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:08:28.662612 2201 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:08:28.662685 2201 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:28.662756 2201 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:08:28.662807 2201 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 04:08:28 dist-test-slave-00ld krb5kdc[2084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222908, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:08:28.673359 2201 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46481
I20260924 04:08:28.673422 2331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46481 every 8 connection(s)
I20260924 04:08:28.674381 2201 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:08:28.676560 2178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:28.663529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56623 (local address 127.0.98.62:35555)
0924 04:08:28.663655 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:28.663659 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:28.664834 (+ 1175us) server_negotiation.cc:407] Connection header received
0924 04:08:28.665958 (+ 1124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:28.665963 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:28.666064 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:28.666131 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:28.667901 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:28.668839 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:28.670291 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:28.670601 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:28.670714 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:28.673120 (+ 2406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:28.673126 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:28.673129 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:28.673158 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:28.674740 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:28.675303 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:28.675306 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:28.675308 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:28.675392 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:28.675679 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:28.675681 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:28.675682 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:28.675866 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:28.675942 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:28.676347 (+ 405us) server_negotiation.cc:299] Negotiation successful
0924 04:08:28.676446 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 04:08:28.677428 2225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:28.664082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35555 (local address 127.0.98.1:56623)
0924 04:08:28.664623 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:28.664665 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:28.665682 (+ 1017us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:28.666574 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:28.666583 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:28.667015 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:08:28.667692 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:28.667712 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:28.668994 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:28.668997 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:28.670173 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:28.670181 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:28.670749 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:28.670753 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:28.670933 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:28.671503 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:08:28.671518 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:08:28.672960 (+ 1442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:08:28.674882 (+ 1922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:28.674887 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:28.674901 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:28.675201 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:28.675474 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:28.675476 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:28.675478 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:28.675598 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:28.675950 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:08:28.675955 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:08:28.676155 (+ 200us) client_negotiation.cc:769] Sending connection context
0924 04:08:28.676295 (+ 140us) client_negotiation.cc:240] Negotiation successful
0924 04:08:28.676521 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":157,"threads_started":1}
I20260924 04:08:28.678644 2332 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35555
I20260924 04:08:28.678790 2332 heartbeater.cc:464] Registering TS with master...
I20260924 04:08:28.679265 2332 heartbeater.cc:511] Master 127.0.98.62:35555 requested a full tablet report, sending...
I20260924 04:08:28.680740 2128 ts_manager.cc:194] Registered new tserver with Master: 8c9d6c009d9642109c86c0cd26db2f50 (127.0.98.1:46481)
I20260924 04:08:28.681571 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 2201
I20260924 04:08:28.681697 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:08:28.681945 392 external_mini_cluster.cc:1428] Setting key f20b039f6e1d887eec3e38113798e5fd
I20260924 04:08:28.682180 2128 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:56623
I20260924 04:08:28.688546 2178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:28.680810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56681 (local address 127.0.98.62:35555)
0924 04:08:28.681117 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:28.681121 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:28.681391 (+ 270us) server_negotiation.cc:407] Connection header received
0924 04:08:28.681440 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:28.681444 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:28.681494 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:28.681582 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:28.682047 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:28.682748 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:28.683849 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:28.684066 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:28.684171 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:28.685449 (+ 1278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:28.685452 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:28.685454 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:28.685474 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:28.686893 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:28.687436 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:28.687439 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:28.687440 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:28.687509 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:28.687802 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:28.687804 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:28.687806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:28.687955 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:28.688026 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:28.688343 (+ 317us) server_negotiation.cc:299] Negotiation successful
0924 04:08:28.688435 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241}
I20260924 04:08:28.688779 2338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:28.680746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35555 (local address 127.0.98.1:56681)
0924 04:08:28.681257 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:28.681271 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:28.681363 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:28.681611 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:28.681614 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:28.681814 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:08:28.681927 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:28.681933 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:28.683103 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:28.683107 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:28.683721 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:28.683730 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:28.684171 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:28.684174 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:28.684265 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:28.684810 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:08:28.684836 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:08:28.685303 (+ 467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:08:28.687031 (+ 1728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:28.687036 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:28.687040 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:28.687311 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:28.687600 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:28.687605 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:28.687607 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:28.687710 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:28.688047 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:08:28.688053 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:08:28.688173 (+ 120us) client_negotiation.cc:769] Sending connection context
0924 04:08:28.688302 (+ 129us) client_negotiation.cc:240] Negotiation successful
0924 04:08:28.688485 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"mutex_wait_us":1,"thread_start_us":108,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:08:28.709280 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35555
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:08:28.819976 2341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:08:28.820228 2341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:08:28.820281 2341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:08:28.824278 2341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:08:28.824335 2341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:08:28.824374 2341 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 04:08:28.824388 2341 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 04:08:28.824419 2341 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:08:28.829571 2341 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:35555
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2341
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:08:28.830550 2341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:08:28.831514 2341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:08:28.841346 2349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:08:28.841362 2347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:08:28.841537 2341 server_base.cc:1034] running on GCE node
W20260924 04:08:28.841364 2346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:08:28.842007 2341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:08:28.842680 2341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:08:28.843875 2341 hybrid_clock.cc:648] HybridClock initialized: now 1790222908843842 us; error 49 us; skew 500 ppm
Sep 24 04:08:28 dist-test-slave-00ld krb5kdc[2084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222908, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:08:28.846647 2341 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:08:28.847504 2341 webserver.cc:460] Webserver started at http://127.0.98.2:43595/ using document root <none> and password file <none>
I20260924 04:08:28.848067 2341 fs_manager.cc:359] Metadata directory not provided
I20260924 04:08:28.848156 2341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:08:28.848384 2341 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:08:28.850426 2341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/minicluster-data/ts-1/data/instance:
uuid: "b223224d054246d796a20ded0415a012"
format_stamp: "Formatted at 2026-09-24 04:08:28 on dist-test-slave-00ld"
server_key: "1b06a37bcea20048531e5bd23c1fb2fb"
server_key_iv: "36dcf10635209d972544b0421073da64"
server_key_version: "encryptionkey@0"
I20260924 04:08:28.850977 2341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b223224d054246d796a20ded0415a012"
format_stamp: "Formatted at 2026-09-24 04:08:28 on dist-test-slave-00ld"
server_key: "1b06a37bcea20048531e5bd23c1fb2fb"
server_key_iv: "36dcf10635209d972544b0421073da64"
server_key_version: "encryptionkey@0"
I20260924 04:08:28.854646 2341 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 04:08:28.857030 2356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:28.858270 2341 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 04:08:28.858409 2341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/minicluster-data/ts-1/wal
uuid: "b223224d054246d796a20ded0415a012"
format_stamp: "Formatted at 2026-09-24 04:08:28 on dist-test-slave-00ld"
server_key: "1b06a37bcea20048531e5bd23c1fb2fb"
server_key_iv: "36dcf10635209d972544b0421073da64"
server_key_version: "encryptionkey@0"
I20260924 04:08:28.858520 2341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:08:28.889036 2341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:08:28.891944 2341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:08:28.892145 2341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:08:28.893653 2341 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:08:28.893733 2341 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:28.893805 2341 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:08:28.893848 2341 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 04:08:28 dist-test-slave-00ld krb5kdc[2084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222908, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:08:28.904711 2341 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34761
I20260924 04:08:28.904789 2471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34761 every 8 connection(s)
I20260924 04:08:28.905733 2341 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:08:28.908007 2178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:28.894588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46201 (local address 127.0.98.62:35555)
0924 04:08:28.894732 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:28.894738 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:28.895655 (+ 917us) server_negotiation.cc:407] Connection header received
0924 04:08:28.896789 (+ 1134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:28.896795 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:28.896854 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:28.896919 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:28.898717 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:28.899414 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:28.901104 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:28.901542 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:28.901690 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:28.904218 (+ 2528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:28.904222 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:28.904225 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:28.904252 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:28.906084 (+ 1832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:28.906643 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:28.906646 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:28.906648 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:28.906735 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:28.907056 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:28.907058 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:28.907059 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:28.907225 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:28.907302 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:28.907798 (+ 496us) server_negotiation.cc:299] Negotiation successful
0924 04:08:28.907893 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 04:08:28.908867 2365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:28.895005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35555 (local address 127.0.98.2:46201)
0924 04:08:28.895489 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:28.895538 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:28.896500 (+ 962us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:28.897366 (+ 866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:28.897377 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:28.897821 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:08:28.898549 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:28.898569 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:28.899539 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:28.899543 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:28.900981 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:28.900994 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:28.901730 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:28.901736 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:28.901962 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:28.902560 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:08:28.902586 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:08:28.904043 (+ 1457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:08:28.906197 (+ 2154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:28.906203 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:28.906218 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:28.906557 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:28.906830 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:28.906834 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:28.906836 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:28.906974 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:28.907346 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:08:28.907351 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:08:28.907586 (+ 235us) client_negotiation.cc:769] Sending connection context
0924 04:08:28.907744 (+ 158us) client_negotiation.cc:240] Negotiation successful
0924 04:08:28.907978 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":188,"threads_started":1}
I20260924 04:08:28.910105 2472 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35555
I20260924 04:08:28.910231 2472 heartbeater.cc:464] Registering TS with master...
I20260924 04:08:28.910755 2472 heartbeater.cc:511] Master 127.0.98.62:35555 requested a full tablet report, sending...
I20260924 04:08:28.911870 2129 ts_manager.cc:194] Registered new tserver with Master: b223224d054246d796a20ded0415a012 (127.0.98.2:34761)
I20260924 04:08:28.912492 2129 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:46201
I20260924 04:08:28.915679 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 2341
I20260924 04:08:28.915789 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:08:28.916119 392 external_mini_cluster.cc:1428] Setting key 312c8951e4882a62793471f8163598d1
I20260924 04:08:28.919577 2478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:28.912092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35555 (local address 127.0.98.2:38367)
0924 04:08:28.912334 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:28.912348 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:28.912448 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:28.912720 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:28.912724 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:28.912933 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:08:28.913027 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:28.913033 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:28.913769 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:28.913772 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:28.914324 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:28.914332 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:28.914787 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:28.914790 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:28.914878 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:28.915425 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:08:28.915443 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:08:28.915878 (+ 435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:08:28.917806 (+ 1928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:28.917809 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:28.917812 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:28.918062 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:28.918323 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:28.918330 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:28.918334 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:28.918475 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:28.918935 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:08:28.918941 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:08:28.919056 (+ 115us) client_negotiation.cc:769] Sending connection context
0924 04:08:28.919174 (+ 118us) client_negotiation.cc:240] Negotiation successful
0924 04:08:28.919335 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":96,"threads_started":1}
I20260924 04:08:28.919577 2178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:28.912352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38367 (local address 127.0.98.62:35555)
0924 04:08:28.912487 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:28.912491 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:28.912506 (+ 15us) server_negotiation.cc:407] Connection header received
0924 04:08:28.912543 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:28.912548 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:28.912591 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:28.912668 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:28.913146 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:28.913642 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:28.914450 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:28.914676 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:28.914787 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:28.916058 (+ 1271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:28.916061 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:28.916063 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:28.916083 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:28.917694 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:28.918153 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:28.918156 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:28.918157 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:28.918224 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:28.918605 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:28.918608 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:28.918610 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:28.918836 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:28.918919 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:28.919223 (+ 304us) server_negotiation.cc:299] Negotiation successful
0924 04:08:28.919379 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:08:28.944289 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35555
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:08:29.056354 2481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:08:29.056645 2481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:08:29.056739 2481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:08:29.060897 2481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:08:29.061002 2481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:08:29.061074 2481 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 04:08:29.061106 2481 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 04:08:29.061174 2481 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:08:29.066602 2481 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:35555
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2481
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:08:29.067629 2481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:08:29.068650 2481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:08:29.078079 2487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:08:29.078126 2481 server_base.cc:1034] running on GCE node
W20260924 04:08:29.078096 2486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:08:29.078091 2489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:08:29.078814 2481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:08:29.079468 2481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:08:29.080654 2481 hybrid_clock.cc:648] HybridClock initialized: now 1790222909080641 us; error 25 us; skew 500 ppm
Sep 24 04:08:29 dist-test-slave-00ld krb5kdc[2084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222909, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:08:29.083449 2481 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:08:29.084280 2481 webserver.cc:460] Webserver started at http://127.0.98.3:44919/ using document root <none> and password file <none>
I20260924 04:08:29.084847 2481 fs_manager.cc:359] Metadata directory not provided
I20260924 04:08:29.084911 2481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:08:29.085072 2481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:08:29.087033 2481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/minicluster-data/ts-2/data/instance:
uuid: "9c5340a31b954a05aa99bb41395ed290"
format_stamp: "Formatted at 2026-09-24 04:08:29 on dist-test-slave-00ld"
server_key: "1aa80b5ab18d14a8a3386efa9be74a50"
server_key_iv: "eeb07e4754c6ef843bd20ac4fbf14d2e"
server_key_version: "encryptionkey@0"
I20260924 04:08:29.087543 2481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9c5340a31b954a05aa99bb41395ed290"
format_stamp: "Formatted at 2026-09-24 04:08:29 on dist-test-slave-00ld"
server_key: "1aa80b5ab18d14a8a3386efa9be74a50"
server_key_iv: "eeb07e4754c6ef843bd20ac4fbf14d2e"
server_key_version: "encryptionkey@0"
I20260924 04:08:29.091128 2481 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 04:08:29.093513 2496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:29.094729 2481 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:08:29.094866 2481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/minicluster-data/ts-2/wal
uuid: "9c5340a31b954a05aa99bb41395ed290"
format_stamp: "Formatted at 2026-09-24 04:08:29 on dist-test-slave-00ld"
server_key: "1aa80b5ab18d14a8a3386efa9be74a50"
server_key_iv: "eeb07e4754c6ef843bd20ac4fbf14d2e"
server_key_version: "encryptionkey@0"
I20260924 04:08:29.094973 2481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:08:29.117945 2481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:08:29.121490 2481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:08:29.121697 2481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:08:29.123176 2481 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:08:29.123224 2481 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:29.123266 2481 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:08:29.123289 2481 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 04:08:29 dist-test-slave-00ld krb5kdc[2084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222909, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:08:29.134182 2481 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34579
I20260924 04:08:29.134367 2611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34579 every 8 connection(s)
I20260924 04:08:29.135180 2481 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:08:29.137059 2178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:29.123977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34063 (local address 127.0.98.62:35555)
0924 04:08:29.124148 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:29.124155 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:29.124999 (+ 844us) server_negotiation.cc:407] Connection header received
0924 04:08:29.125986 (+ 987us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:29.125990 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:29.126045 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:29.126104 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:29.127578 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:29.128161 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:29.129509 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:29.129727 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:29.129837 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:29.133148 (+ 3311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:29.133152 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:29.133155 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:29.133181 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:29.134819 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:29.135430 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:29.135433 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:29.135435 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:29.135520 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:29.135890 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:29.135893 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:29.135895 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:29.136155 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:29.136255 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:29.136763 (+ 508us) server_negotiation.cc:299] Negotiation successful
0924 04:08:29.136888 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 04:08:29.137897 2505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:29.124351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35555 (local address 127.0.98.3:34063)
0924 04:08:29.124836 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:29.124887 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:29.125727 (+ 840us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:29.126564 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:29.126571 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:29.126902 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:08:29.127373 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:29.127386 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:29.128310 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:29.128314 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:29.129379 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:29.129389 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:29.129888 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:29.129892 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:29.130096 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:29.130817 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:08:29.130835 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:08:29.132976 (+ 2141us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:08:29.134936 (+ 1960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:29.134942 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:29.134957 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:29.135318 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:29.135613 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:29.135618 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:29.135620 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:29.135761 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:29.136250 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:08:29.136256 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:08:29.136506 (+ 250us) client_negotiation.cc:769] Sending connection context
0924 04:08:29.136688 (+ 182us) client_negotiation.cc:240] Negotiation successful
0924 04:08:29.137030 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":135,"threads_started":1}
I20260924 04:08:29.139024 2612 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35555
I20260924 04:08:29.139185 2612 heartbeater.cc:464] Registering TS with master...
I20260924 04:08:29.139709 2612 heartbeater.cc:511] Master 127.0.98.62:35555 requested a full tablet report, sending...
I20260924 04:08:29.140810 2128 ts_manager.cc:194] Registered new tserver with Master: 9c5340a31b954a05aa99bb41395ed290 (127.0.98.3:34579)
I20260924 04:08:29.141018 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 2481
I20260924 04:08:29.141114 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:08:29.141394 392 external_mini_cluster.cc:1428] Setting key 308221709ba73e82891244d0b1cd607a
I20260924 04:08:29.141608 2128 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34063
I20260924 04:08:29.143436 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:08:29.149705 2178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:29.141943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59927 (local address 127.0.98.62:35555)
0924 04:08:29.142049 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:29.142053 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:29.142065 (+ 12us) server_negotiation.cc:407] Connection header received
0924 04:08:29.142098 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:29.142102 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:29.142145 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:29.142242 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:29.142684 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:29.143232 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:29.144402 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:29.144606 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:29.144754 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:29.146027 (+ 1273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:29.146034 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:29.146039 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:29.146068 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:29.147767 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:29.148293 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:29.148300 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:29.148305 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:29.148437 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:29.148842 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:29.148845 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:29.148847 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:29.149086 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:29.149189 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:29.149496 (+ 307us) server_negotiation.cc:299] Negotiation successful
0924 04:08:29.149590 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 04:08:29.149936 2618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:29.141128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35555 (local address 127.0.98.3:59927)
0924 04:08:29.141367 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:29.141383 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:29.141455 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:29.142277 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:29.142280 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:29.142464 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:08:29.142575 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:29.142581 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:29.143394 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:29.143397 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:29.144280 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:29.144295 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:29.144778 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:29.144784 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:29.144918 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:29.145503 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:08:29.145524 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:08:29.145866 (+ 342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:08:29.147895 (+ 2029us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:29.147898 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:29.147899 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:29.148165 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:29.148562 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:29.148565 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:29.148566 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:29.148646 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:29.149210 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:08:29.149214 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:08:29.149321 (+ 107us) client_negotiation.cc:769] Sending connection context
0924 04:08:29.149442 (+ 121us) client_negotiation.cc:240] Negotiation successful
0924 04:08:29.149605 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":1,"thread_start_us":103,"threads_started":1}
Sep 24 04:08:29 dist-test-slave-00ld krb5kdc[2084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222909, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 04:08:29 dist-test-slave-00ld krb5kdc[2084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222909, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:08:29.171229 2178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:29.162836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53412 (local address 127.0.98.62:35555)
0924 04:08:29.162961 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:29.162964 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:29.162978 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:08:29.163021 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:29.163023 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:29.163063 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:29.163134 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:29.163879 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:29.164473 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:29.165431 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:29.165657 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:29.165739 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:29.168373 (+ 2634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:29.168376 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:29.168378 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:29.168403 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:29.169722 (+ 1319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:29.170143 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:29.170145 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:29.170146 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:29.170218 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:29.170481 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:29.170483 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:29.170485 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:29.170648 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:29.170746 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:29.170935 (+ 189us) server_negotiation.cc:299] Negotiation successful
0924 04:08:29.171030 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 04:08:29.174315 2128 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53412:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:08:29.177438 2128 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:08:29.191919 2634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:29.186513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56744 (local address 127.0.98.2:34761)
0924 04:08:29.186927 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:29.186940 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:29.187377 (+ 437us) server_negotiation.cc:407] Connection header received
0924 04:08:29.187442 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:29.187446 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:29.187590 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:29.187744 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:08:29.187958 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:29.189128 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:29.190664 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:29.191228 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:29.191371 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:29.191548 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:29.191664 (+ 116us) server_negotiation.cc:299] Negotiation successful
0924 04:08:29.191756 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":104,"threads_started":1}
I20260924 04:08:29.191900 2633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:29.186343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34761 (local address 127.0.0.1:56744)
0924 04:08:29.187171 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:29.187185 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:29.187277 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:29.187716 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:29.187719 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:29.187736 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:08:29.187826 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:29.187833 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:29.189261 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:29.189271 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:29.190543 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:29.190552 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:29.191367 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:29.191371 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:29.191478 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:29.191513 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 04:08:29.191633 (+ 120us) client_negotiation.cc:240] Negotiation successful
0924 04:08:29.191756 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":763,"thread_start_us":83,"threads_started":1}
I20260924 04:08:29.193374 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:29.185559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34579 (local address 127.0.0.1:47332)
0924 04:08:29.186428 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:29.186481 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:29.186625 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:29.187372 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:29.187378 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:29.187410 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:08:29.187758 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:29.187763 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:29.190454 (+ 2691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:29.190457 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:29.191625 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:29.191636 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:29.192943 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:29.192947 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:29.193055 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:29.193085 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 04:08:29.193158 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 04:08:29.193222 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":743,"mutex_wait_us":139,"spinlock_wait_cycles":9728,"thread_start_us":87,"threads_started":1}
I20260924 04:08:29.193722 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:29.185559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46481 (local address 127.0.0.1:53524)
0924 04:08:29.186428 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:29.186481 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:29.186625 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:29.187372 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:29.187378 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:29.187411 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:08:29.187749 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:29.187757 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:29.190674 (+ 2917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:29.190677 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:29.192054 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:29.192063 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:29.193345 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:29.193350 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:29.193446 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:29.193478 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 04:08:29.193548 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 04:08:29.193612 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":659,"mutex_wait_us":88,"spinlock_wait_cycles":10240,"thread_start_us":94,"threads_started":1}
I20260924 04:08:29.193828 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:29.185713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47332 (local address 127.0.98.3:34579)
0924 04:08:29.186146 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:29.186157 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:29.186776 (+ 619us) server_negotiation.cc:407] Connection header received
0924 04:08:29.186853 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:29.186858 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:29.187066 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:29.187209 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:08:29.189291 (+ 2082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:29.190344 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:29.191783 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:29.192552 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:29.192660 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:29.192844 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:29.193602 (+ 758us) server_negotiation.cc:299] Negotiation successful
0924 04:08:29.193696 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":113,"threads_started":1}
I20260924 04:08:29.194679 2632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:29.185856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53524 (local address 127.0.98.1:46481)
0924 04:08:29.186146 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:29.186155 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:29.186771 (+ 616us) server_negotiation.cc:407] Connection header received
0924 04:08:29.186853 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:29.186858 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:29.187066 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:29.187210 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:08:29.187905 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:29.189159 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:29.192189 (+ 3030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:29.193023 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:29.193104 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:29.193253 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:29.194505 (+ 1252us) server_negotiation.cc:299] Negotiation successful
0924 04:08:29.194584 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":70,"threads_started":1}
I20260924 04:08:29.195291 2406 tablet_service.cc:1467] Processing CreateTablet for tablet 435cbf6ae4754312bcf1858b6ad9c57b (DEFAULT_TABLE table=test-table [id=a7faa8c8be254badb078a9509a1f587a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:08:29.196439 2406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 435cbf6ae4754312bcf1858b6ad9c57b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:08:29.196410 2546 tablet_service.cc:1467] Processing CreateTablet for tablet 435cbf6ae4754312bcf1858b6ad9c57b (DEFAULT_TABLE table=test-table [id=a7faa8c8be254badb078a9509a1f587a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:08:29.196844 2266 tablet_service.cc:1467] Processing CreateTablet for tablet 435cbf6ae4754312bcf1858b6ad9c57b (DEFAULT_TABLE table=test-table [id=a7faa8c8be254badb078a9509a1f587a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:08:29.197448 2546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 435cbf6ae4754312bcf1858b6ad9c57b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:08:29.197734 2266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 435cbf6ae4754312bcf1858b6ad9c57b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:08:29.204483 2636 tablet_bootstrap.cc:492] T 435cbf6ae4754312bcf1858b6ad9c57b P b223224d054246d796a20ded0415a012: Bootstrap starting.
I20260924 04:08:29.205673 2635 tablet_bootstrap.cc:492] T 435cbf6ae4754312bcf1858b6ad9c57b P 8c9d6c009d9642109c86c0cd26db2f50: Bootstrap starting.
I20260924 04:08:29.206174 2637 tablet_bootstrap.cc:492] T 435cbf6ae4754312bcf1858b6ad9c57b P 9c5340a31b954a05aa99bb41395ed290: Bootstrap starting.
I20260924 04:08:29.207172 2636 tablet_bootstrap.cc:654] T 435cbf6ae4754312bcf1858b6ad9c57b P b223224d054246d796a20ded0415a012: Neither blocks nor log segments found. Creating new log.
I20260924 04:08:29.207805 2635 tablet_bootstrap.cc:654] T 435cbf6ae4754312bcf1858b6ad9c57b P 8c9d6c009d9642109c86c0cd26db2f50: Neither blocks nor log segments found. Creating new log.
I20260924 04:08:29.207897 2637 tablet_bootstrap.cc:654] T 435cbf6ae4754312bcf1858b6ad9c57b P 9c5340a31b954a05aa99bb41395ed290: Neither blocks nor log segments found. Creating new log.
I20260924 04:08:29.208107 2636 log.cc:824] T 435cbf6ae4754312bcf1858b6ad9c57b P b223224d054246d796a20ded0415a012: Log is configured to *not* fsync() on all Append() calls
I20260924 04:08:29.208585 2635 log.cc:824] T 435cbf6ae4754312bcf1858b6ad9c57b P 8c9d6c009d9642109c86c0cd26db2f50: Log is configured to *not* fsync() on all Append() calls
I20260924 04:08:29.208704 2637 log.cc:824] T 435cbf6ae4754312bcf1858b6ad9c57b P 9c5340a31b954a05aa99bb41395ed290: Log is configured to *not* fsync() on all Append() calls
I20260924 04:08:29.210410 2636 tablet_bootstrap.cc:492] T 435cbf6ae4754312bcf1858b6ad9c57b P b223224d054246d796a20ded0415a012: No bootstrap required, opened a new log
I20260924 04:08:29.210593 2635 tablet_bootstrap.cc:492] T 435cbf6ae4754312bcf1858b6ad9c57b P 8c9d6c009d9642109c86c0cd26db2f50: No bootstrap required, opened a new log
I20260924 04:08:29.210606 2636 ts_tablet_manager.cc:1397] T 435cbf6ae4754312bcf1858b6ad9c57b P b223224d054246d796a20ded0415a012: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 04:08:29.210767 2635 ts_tablet_manager.cc:1397] T 435cbf6ae4754312bcf1858b6ad9c57b P 8c9d6c009d9642109c86c0cd26db2f50: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 04:08:29.213737 2637 tablet_bootstrap.cc:492] T 435cbf6ae4754312bcf1858b6ad9c57b P 9c5340a31b954a05aa99bb41395ed290: No bootstrap required, opened a new log
I20260924 04:08:29.213941 2637 ts_tablet_manager.cc:1397] T 435cbf6ae4754312bcf1858b6ad9c57b P 9c5340a31b954a05aa99bb41395ed290: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20260924 04:08:29.214682 2636 raft_consensus.cc:348] T 435cbf6ae4754312bcf1858b6ad9c57b P b223224d054246d796a20ded0415a012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c5340a31b954a05aa99bb41395ed290" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34579 } } peers { permanent_uuid: "8c9d6c009d9642109c86c0cd26db2f50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46481 } } peers { permanent_uuid: "b223224d054246d796a20ded0415a012" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34761 } }
I20260924 04:08:29.214957 2636 raft_consensus.cc:374] T 435cbf6ae4754312bcf1858b6ad9c57b P b223224d054246d796a20ded0415a012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:08:29.215011 2636 raft_consensus.cc:729] T 435cbf6ae4754312bcf1858b6ad9c57b P b223224d054246d796a20ded0415a012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b223224d054246d796a20ded0415a012, State: Initialized, Role: FOLLOWER
I20260924 04:08:29.214928 2635 raft_consensus.cc:348] T 435cbf6ae4754312bcf1858b6ad9c57b P 8c9d6c009d9642109c86c0cd26db2f50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c5340a31b954a05aa99bb41395ed290" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34579 } } peers { permanent_uuid: "8c9d6c009d9642109c86c0cd26db2f50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46481 } } peers { permanent_uuid: "b223224d054246d796a20ded0415a012" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34761 } }
I20260924 04:08:29.215159 2635 raft_consensus.cc:374] T 435cbf6ae4754312bcf1858b6ad9c57b P 8c9d6c009d9642109c86c0cd26db2f50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:08:29.215214 2635 raft_consensus.cc:729] T 435cbf6ae4754312bcf1858b6ad9c57b P 8c9d6c009d9642109c86c0cd26db2f50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c9d6c009d9642109c86c0cd26db2f50, State: Initialized, Role: FOLLOWER
I20260924 04:08:29.215466 2636 consensus_queue.cc:260] T 435cbf6ae4754312bcf1858b6ad9c57b P b223224d054246d796a20ded0415a012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c5340a31b954a05aa99bb41395ed290" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34579 } } peers { permanent_uuid: "8c9d6c009d9642109c86c0cd26db2f50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46481 } } peers { permanent_uuid: "b223224d054246d796a20ded0415a012" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34761 } }
I20260924 04:08:29.215559 2635 consensus_queue.cc:260] T 435cbf6ae4754312bcf1858b6ad9c57b P 8c9d6c009d9642109c86c0cd26db2f50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c5340a31b954a05aa99bb41395ed290" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34579 } } peers { permanent_uuid: "8c9d6c009d9642109c86c0cd26db2f50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46481 } } peers { permanent_uuid: "b223224d054246d796a20ded0415a012" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34761 } }
I20260924 04:08:29.216118 2472 heartbeater.cc:503] Master 127.0.98.62:35555 was elected leader, sending a full tablet report...
I20260924 04:08:29.216795 2636 ts_tablet_manager.cc:1428] T 435cbf6ae4754312bcf1858b6ad9c57b P b223224d054246d796a20ded0415a012: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260924 04:08:29.217254 2635 ts_tablet_manager.cc:1428] T 435cbf6ae4754312bcf1858b6ad9c57b P 8c9d6c009d9642109c86c0cd26db2f50: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 04:08:29.217607 2332 heartbeater.cc:503] Master 127.0.98.62:35555 was elected leader, sending a full tablet report...
I20260924 04:08:29.217595 2637 raft_consensus.cc:348] T 435cbf6ae4754312bcf1858b6ad9c57b P 9c5340a31b954a05aa99bb41395ed290 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c5340a31b954a05aa99bb41395ed290" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34579 } } peers { permanent_uuid: "8c9d6c009d9642109c86c0cd26db2f50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46481 } } peers { permanent_uuid: "b223224d054246d796a20ded0415a012" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34761 } }
I20260924 04:08:29.217813 2637 raft_consensus.cc:374] T 435cbf6ae4754312bcf1858b6ad9c57b P 9c5340a31b954a05aa99bb41395ed290 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:08:29.217847 2637 raft_consensus.cc:729] T 435cbf6ae4754312bcf1858b6ad9c57b P 9c5340a31b954a05aa99bb41395ed290 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c5340a31b954a05aa99bb41395ed290, State: Initialized, Role: FOLLOWER
I20260924 04:08:29.218190 2637 consensus_queue.cc:260] T 435cbf6ae4754312bcf1858b6ad9c57b P 9c5340a31b954a05aa99bb41395ed290 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c5340a31b954a05aa99bb41395ed290" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34579 } } peers { permanent_uuid: "8c9d6c009d9642109c86c0cd26db2f50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46481 } } peers { permanent_uuid: "b223224d054246d796a20ded0415a012" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34761 } }
I20260924 04:08:29.218773 2612 heartbeater.cc:503] Master 127.0.98.62:35555 was elected leader, sending a full tablet report...
I20260924 04:08:29.219871 2637 ts_tablet_manager.cc:1428] T 435cbf6ae4754312bcf1858b6ad9c57b P 9c5340a31b954a05aa99bb41395ed290: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 04:08:29.272472 2643 raft_consensus.cc:482] T 435cbf6ae4754312bcf1858b6ad9c57b P 9c5340a31b954a05aa99bb41395ed290 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:08:29.272658 2643 raft_consensus.cc:504] T 435cbf6ae4754312bcf1858b6ad9c57b P 9c5340a31b954a05aa99bb41395ed290 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c5340a31b954a05aa99bb41395ed290" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34579 } } peers { permanent_uuid: "8c9d6c009d9642109c86c0cd26db2f50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46481 } } peers { permanent_uuid: "b223224d054246d796a20ded0415a012" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34761 } }
I20260924 04:08:29.273838 2643 leader_election.cc:288] T 435cbf6ae4754312bcf1858b6ad9c57b P 9c5340a31b954a05aa99bb41395ed290 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c9d6c009d9642109c86c0cd26db2f50 (127.0.98.1:46481), b223224d054246d796a20ded0415a012 (127.0.98.2:34761)
I20260924 04:08:29.278079 2634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:29.274542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46763 (local address 127.0.98.2:34761)
0924 04:08:29.274663 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:29.274666 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:29.274818 (+ 152us) server_negotiation.cc:407] Connection header received
0924 04:08:29.274958 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:29.274961 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:29.275020 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:29.275096 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:08:29.275398 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:29.275949 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:29.277038 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:29.277531 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:29.277608 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:29.277656 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:29.277875 (+ 219us) server_negotiation.cc:299] Negotiation successful
0924 04:08:29.277953 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 04:08:29.278096 2644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:29.274311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34761 (local address 127.0.98.3:46763)
0924 04:08:29.274733 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:29.274746 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:29.274854 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:29.275114 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:29.275117 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:29.275134 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:08:29.275259 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:29.275265 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:29.276059 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:29.276061 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:29.276928 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:29.276937 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:29.277638 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:29.277642 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:29.277745 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:29.277770 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 04:08:29.277850 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 04:08:29.277932 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"mutex_wait_us":1,"spinlock_wait_cycles":22016,"thread_start_us":128,"threads_started":1}
I20260924 04:08:29.278393 2505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:29.274022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46481 (local address 127.0.98.3:56379)
0924 04:08:29.274336 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:29.274352 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:29.274451 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:29.274763 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:29.274765 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:29.274785 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:08:29.274960 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:29.274966 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:29.275808 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:29.275811 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:29.277221 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:29.277230 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:29.277976 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:29.277979 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:29.278074 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:29.278105 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 04:08:29.278179 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 04:08:29.278272 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"mutex_wait_us":21,"spinlock_wait_cycles":896}
I20260924 04:08:29.278620 2632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:29.274186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56379 (local address 127.0.98.1:46481)
0924 04:08:29.274333 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:29.274338 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:29.274543 (+ 205us) server_negotiation.cc:407] Connection header received
0924 04:08:29.274606 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:29.274608 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:29.274659 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:29.274735 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:08:29.275106 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:29.275706 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:29.277340 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:29.277871 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:29.277962 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:29.278016 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:29.278413 (+ 397us) server_negotiation.cc:299] Negotiation successful
0924 04:08:29.278491 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 04:08:29.278730 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "435cbf6ae4754312bcf1858b6ad9c57b" candidate_uuid: "9c5340a31b954a05aa99bb41395ed290" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b223224d054246d796a20ded0415a012" is_pre_election: true
I20260924 04:08:29.279076 2419 raft_consensus.cc:2454] T 435cbf6ae4754312bcf1858b6ad9c57b P b223224d054246d796a20ded0415a012 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c5340a31b954a05aa99bb41395ed290 in term 0.
I20260924 04:08:29.279117 2283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "435cbf6ae4754312bcf1858b6ad9c57b" candidate_uuid: "9c5340a31b954a05aa99bb41395ed290" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c9d6c009d9642109c86c0cd26db2f50" is_pre_election: true
I20260924 04:08:29.279426 2283 raft_consensus.cc:2454] T 435cbf6ae4754312bcf1858b6ad9c57b P 8c9d6c009d9642109c86c0cd26db2f50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c5340a31b954a05aa99bb41395ed290 in term 0.
I20260924 04:08:29.279538 2497 leader_election.cc:302] T 435cbf6ae4754312bcf1858b6ad9c57b P 9c5340a31b954a05aa99bb41395ed290 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c5340a31b954a05aa99bb41395ed290, b223224d054246d796a20ded0415a012; no voters:
I20260924 04:08:29.279808 2643 raft_consensus.cc:2781] T 435cbf6ae4754312bcf1858b6ad9c57b P 9c5340a31b954a05aa99bb41395ed290 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:08:29.279904 2643 raft_consensus.cc:482] T 435cbf6ae4754312bcf1858b6ad9c57b P 9c5340a31b954a05aa99bb41395ed290 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:08:29.280019 2643 raft_consensus.cc:3037] T 435cbf6ae4754312bcf1858b6ad9c57b P 9c5340a31b954a05aa99bb41395ed290 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:08:29.281064 2643 raft_consensus.cc:504] T 435cbf6ae4754312bcf1858b6ad9c57b P 9c5340a31b954a05aa99bb41395ed290 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c5340a31b954a05aa99bb41395ed290" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34579 } } peers { permanent_uuid: "8c9d6c009d9642109c86c0cd26db2f50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46481 } } peers { permanent_uuid: "b223224d054246d796a20ded0415a012" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34761 } }
I20260924 04:08:29.281395 2643 leader_election.cc:288] T 435cbf6ae4754312bcf1858b6ad9c57b P 9c5340a31b954a05aa99bb41395ed290 [CANDIDATE]: Term 1 election: Requested vote from peers 8c9d6c009d9642109c86c0cd26db2f50 (127.0.98.1:46481), b223224d054246d796a20ded0415a012 (127.0.98.2:34761)
I20260924 04:08:29.281755 2283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "435cbf6ae4754312bcf1858b6ad9c57b" candidate_uuid: "9c5340a31b954a05aa99bb41395ed290" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c9d6c009d9642109c86c0cd26db2f50"
I20260924 04:08:29.281773 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "435cbf6ae4754312bcf1858b6ad9c57b" candidate_uuid: "9c5340a31b954a05aa99bb41395ed290" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b223224d054246d796a20ded0415a012"
I20260924 04:08:29.281872 2283 raft_consensus.cc:3037] T 435cbf6ae4754312bcf1858b6ad9c57b P 8c9d6c009d9642109c86c0cd26db2f50 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:08:29.281891 2419 raft_consensus.cc:3037] T 435cbf6ae4754312bcf1858b6ad9c57b P b223224d054246d796a20ded0415a012 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:08:29.283007 2419 raft_consensus.cc:2454] T 435cbf6ae4754312bcf1858b6ad9c57b P b223224d054246d796a20ded0415a012 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c5340a31b954a05aa99bb41395ed290 in term 1.
I20260924 04:08:29.283129 2283 raft_consensus.cc:2454] T 435cbf6ae4754312bcf1858b6ad9c57b P 8c9d6c009d9642109c86c0cd26db2f50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c5340a31b954a05aa99bb41395ed290 in term 1.
I20260924 04:08:29.283329 2497 leader_election.cc:302] T 435cbf6ae4754312bcf1858b6ad9c57b P 9c5340a31b954a05aa99bb41395ed290 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c5340a31b954a05aa99bb41395ed290, b223224d054246d796a20ded0415a012; no voters:
I20260924 04:08:29.283572 2643 raft_consensus.cc:2781] T 435cbf6ae4754312bcf1858b6ad9c57b P 9c5340a31b954a05aa99bb41395ed290 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:08:29.283809 2643 raft_consensus.cc:686] T 435cbf6ae4754312bcf1858b6ad9c57b P 9c5340a31b954a05aa99bb41395ed290 [term 1 LEADER]: Becoming Leader. State: Replica: 9c5340a31b954a05aa99bb41395ed290, State: Running, Role: LEADER
I20260924 04:08:29.284144 2643 consensus_queue.cc:237] T 435cbf6ae4754312bcf1858b6ad9c57b P 9c5340a31b954a05aa99bb41395ed290 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c5340a31b954a05aa99bb41395ed290" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34579 } } peers { permanent_uuid: "8c9d6c009d9642109c86c0cd26db2f50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46481 } } peers { permanent_uuid: "b223224d054246d796a20ded0415a012" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34761 } }
I20260924 04:08:29.287243 2127 catalog_manager.cc:5495] T 435cbf6ae4754312bcf1858b6ad9c57b P 9c5340a31b954a05aa99bb41395ed290 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c5340a31b954a05aa99bb41395ed290 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "9c5340a31b954a05aa99bb41395ed290" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c5340a31b954a05aa99bb41395ed290" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34579 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c9d6c009d9642109c86c0cd26db2f50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b223224d054246d796a20ded0415a012" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34761 } health_report { overall_health: UNKNOWN } } }
Sep 24 04:08:29 dist-test-slave-00ld krb5kdc[2084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222908, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:08:29.339113 2178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:29.330511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53428 (local address 127.0.98.62:35555)
0924 04:08:29.330676 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:29.330681 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:29.330849 (+ 168us) server_negotiation.cc:407] Connection header received
0924 04:08:29.331025 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:29.331027 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:29.331061 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:29.331140 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:29.331811 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:29.332333 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:29.333181 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:29.333393 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:29.333499 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:29.335935 (+ 2436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:29.335939 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:29.335942 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:29.335968 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:29.337213 (+ 1245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:29.337665 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:29.337671 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:29.337673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:29.337752 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:29.338019 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:29.338022 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:29.338023 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:29.338187 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:29.338303 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:29.338667 (+ 364us) server_negotiation.cc:299] Negotiation successful
0924 04:08:29.338820 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 04:08:29.339118 2652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:29.330452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35555 (local address 127.0.0.1:53428)
0924 04:08:29.330758 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:29.330770 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:29.330905 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:29.331165 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:29.331168 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:29.331589 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:08:29.331675 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:29.331681 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:29.332437 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:29.332444 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:29.333069 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:29.333077 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:29.333500 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:29.333504 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:29.333580 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:29.334192 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:08:29.334203 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:08:29.335812 (+ 1609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:08:29.337318 (+ 1506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:29.337329 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:29.337335 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:29.337567 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:29.337843 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:29.337846 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:29.337848 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:29.337926 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:29.338302 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:08:29.338309 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:08:29.338432 (+ 123us) client_negotiation.cc:769] Sending connection context
0924 04:08:29.338551 (+ 119us) client_negotiation.cc:240] Negotiation successful
0924 04:08:29.338723 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":85,"threads_started":1}
I20260924 04:08:29.344426 2127 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53428:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 04:08:29.345304 2127 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:08:29.350231 2266 tablet_service.cc:1467] Processing CreateTablet for tablet 09195344e10140438ec7945c4f1cb598 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=99cc6f61b77c4821881304741f327a81]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 04:08:29.350320 2546 tablet_service.cc:1467] Processing CreateTablet for tablet 09195344e10140438ec7945c4f1cb598 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=99cc6f61b77c4821881304741f327a81]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 04:08:29.350488 2406 tablet_service.cc:1467] Processing CreateTablet for tablet 09195344e10140438ec7945c4f1cb598 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=99cc6f61b77c4821881304741f327a81]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 04:08:29.350952 2266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09195344e10140438ec7945c4f1cb598. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:08:29.351023 2406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09195344e10140438ec7945c4f1cb598. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:08:29.351023 2546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09195344e10140438ec7945c4f1cb598. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:08:29.353902 2637 tablet_bootstrap.cc:492] T 09195344e10140438ec7945c4f1cb598 P 9c5340a31b954a05aa99bb41395ed290: Bootstrap starting.
I20260924 04:08:29.353912 2635 tablet_bootstrap.cc:492] T 09195344e10140438ec7945c4f1cb598 P 8c9d6c009d9642109c86c0cd26db2f50: Bootstrap starting.
I20260924 04:08:29.354732 2637 tablet_bootstrap.cc:654] T 09195344e10140438ec7945c4f1cb598 P 9c5340a31b954a05aa99bb41395ed290: Neither blocks nor log segments found. Creating new log.
I20260924 04:08:29.354899 2635 tablet_bootstrap.cc:654] T 09195344e10140438ec7945c4f1cb598 P 8c9d6c009d9642109c86c0cd26db2f50: Neither blocks nor log segments found. Creating new log.
I20260924 04:08:29.355480 2636 tablet_bootstrap.cc:492] T 09195344e10140438ec7945c4f1cb598 P b223224d054246d796a20ded0415a012: Bootstrap starting.
I20260924 04:08:29.355870 2637 tablet_bootstrap.cc:492] T 09195344e10140438ec7945c4f1cb598 P 9c5340a31b954a05aa99bb41395ed290: No bootstrap required, opened a new log
I20260924 04:08:29.355940 2637 ts_tablet_manager.cc:1397] T 09195344e10140438ec7945c4f1cb598 P 9c5340a31b954a05aa99bb41395ed290: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 04:08:29.356389 2636 tablet_bootstrap.cc:654] T 09195344e10140438ec7945c4f1cb598 P b223224d054246d796a20ded0415a012: Neither blocks nor log segments found. Creating new log.
I20260924 04:08:29.356387 2637 raft_consensus.cc:348] T 09195344e10140438ec7945c4f1cb598 P 9c5340a31b954a05aa99bb41395ed290 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c9d6c009d9642109c86c0cd26db2f50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46481 } } peers { permanent_uuid: "b223224d054246d796a20ded0415a012" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34761 } } peers { permanent_uuid: "9c5340a31b954a05aa99bb41395ed290" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34579 } }
I20260924 04:08:29.356494 2637 raft_consensus.cc:374] T 09195344e10140438ec7945c4f1cb598 P 9c5340a31b954a05aa99bb41395ed290 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:08:29.356528 2637 raft_consensus.cc:729] T 09195344e10140438ec7945c4f1cb598 P 9c5340a31b954a05aa99bb41395ed290 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c5340a31b954a05aa99bb41395ed290, State: Initialized, Role: FOLLOWER
I20260924 04:08:29.356652 2637 consensus_queue.cc:260] T 09195344e10140438ec7945c4f1cb598 P 9c5340a31b954a05aa99bb41395ed290 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c9d6c009d9642109c86c0cd26db2f50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46481 } } peers { permanent_uuid: "b223224d054246d796a20ded0415a012" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34761 } } peers { permanent_uuid: "9c5340a31b954a05aa99bb41395ed290" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34579 } }
I20260924 04:08:29.356972 2637 ts_tablet_manager.cc:1428] T 09195344e10140438ec7945c4f1cb598 P 9c5340a31b954a05aa99bb41395ed290: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:08:29.357117 2643 tablet_replica.cc:439] T 09195344e10140438ec7945c4f1cb598 P 9c5340a31b954a05aa99bb41395ed290: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c9d6c009d9642109c86c0cd26db2f50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46481 } } peers { permanent_uuid: "b223224d054246d796a20ded0415a012" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34761 } } peers { permanent_uuid: "9c5340a31b954a05aa99bb41395ed290" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34579 } } }
I20260924 04:08:29.357280 2643 tablet_replica.cc:442] T 09195344e10140438ec7945c4f1cb598 P 9c5340a31b954a05aa99bb41395ed290: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:08:29.357522 2635 tablet_bootstrap.cc:492] T 09195344e10140438ec7945c4f1cb598 P 8c9d6c009d9642109c86c0cd26db2f50: No bootstrap required, opened a new log
I20260924 04:08:29.357625 2635 ts_tablet_manager.cc:1397] T 09195344e10140438ec7945c4f1cb598 P 8c9d6c009d9642109c86c0cd26db2f50: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260924 04:08:29.358060 2636 tablet_bootstrap.cc:492] T 09195344e10140438ec7945c4f1cb598 P b223224d054246d796a20ded0415a012: No bootstrap required, opened a new log
I20260924 04:08:29.358124 2636 ts_tablet_manager.cc:1397] T 09195344e10140438ec7945c4f1cb598 P b223224d054246d796a20ded0415a012: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 04:08:29.358124 2635 raft_consensus.cc:348] T 09195344e10140438ec7945c4f1cb598 P 8c9d6c009d9642109c86c0cd26db2f50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c9d6c009d9642109c86c0cd26db2f50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46481 } } peers { permanent_uuid: "b223224d054246d796a20ded0415a012" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34761 } } peers { permanent_uuid: "9c5340a31b954a05aa99bb41395ed290" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34579 } }
I20260924 04:08:29.358247 2635 raft_consensus.cc:374] T 09195344e10140438ec7945c4f1cb598 P 8c9d6c009d9642109c86c0cd26db2f50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:08:29.358291 2635 raft_consensus.cc:729] T 09195344e10140438ec7945c4f1cb598 P 8c9d6c009d9642109c86c0cd26db2f50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c9d6c009d9642109c86c0cd26db2f50, State: Initialized, Role: FOLLOWER
I20260924 04:08:29.358428 2635 consensus_queue.cc:260] T 09195344e10140438ec7945c4f1cb598 P 8c9d6c009d9642109c86c0cd26db2f50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c9d6c009d9642109c86c0cd26db2f50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46481 } } peers { permanent_uuid: "b223224d054246d796a20ded0415a012" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34761 } } peers { permanent_uuid: "9c5340a31b954a05aa99bb41395ed290" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34579 } }
I20260924 04:08:29.358520 2636 raft_consensus.cc:348] T 09195344e10140438ec7945c4f1cb598 P b223224d054246d796a20ded0415a012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c9d6c009d9642109c86c0cd26db2f50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46481 } } peers { permanent_uuid: "b223224d054246d796a20ded0415a012" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34761 } } peers { permanent_uuid: "9c5340a31b954a05aa99bb41395ed290" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34579 } }
I20260924 04:08:29.358637 2636 raft_consensus.cc:374] T 09195344e10140438ec7945c4f1cb598 P b223224d054246d796a20ded0415a012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:08:29.358671 2636 raft_consensus.cc:729] T 09195344e10140438ec7945c4f1cb598 P b223224d054246d796a20ded0415a012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b223224d054246d796a20ded0415a012, State: Initialized, Role: FOLLOWER
I20260924 04:08:29.358784 2636 consensus_queue.cc:260] T 09195344e10140438ec7945c4f1cb598 P b223224d054246d796a20ded0415a012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c9d6c009d9642109c86c0cd26db2f50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46481 } } peers { permanent_uuid: "b223224d054246d796a20ded0415a012" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34761 } } peers { permanent_uuid: "9c5340a31b954a05aa99bb41395ed290" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34579 } }
I20260924 04:08:29.358727 2642 tablet_replica.cc:439] T 09195344e10140438ec7945c4f1cb598 P 8c9d6c009d9642109c86c0cd26db2f50: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c9d6c009d9642109c86c0cd26db2f50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46481 } } peers { permanent_uuid: "b223224d054246d796a20ded0415a012" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34761 } } peers { permanent_uuid: "9c5340a31b954a05aa99bb41395ed290" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34579 } } }
I20260924 04:08:29.358896 2642 tablet_replica.cc:442] T 09195344e10140438ec7945c4f1cb598 P 8c9d6c009d9642109c86c0cd26db2f50: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:08:29.358716 2635 ts_tablet_manager.cc:1428] T 09195344e10140438ec7945c4f1cb598 P 8c9d6c009d9642109c86c0cd26db2f50: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 04:08:29.359028 2636 ts_tablet_manager.cc:1428] T 09195344e10140438ec7945c4f1cb598 P b223224d054246d796a20ded0415a012: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:08:29.359238 2641 tablet_replica.cc:439] T 09195344e10140438ec7945c4f1cb598 P b223224d054246d796a20ded0415a012: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c9d6c009d9642109c86c0cd26db2f50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46481 } } peers { permanent_uuid: "b223224d054246d796a20ded0415a012" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34761 } } peers { permanent_uuid: "9c5340a31b954a05aa99bb41395ed290" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34579 } } }
I20260924 04:08:29.359367 2641 tablet_replica.cc:442] T 09195344e10140438ec7945c4f1cb598 P b223224d054246d796a20ded0415a012: This TxnStatusTablet replica's current role is: FOLLOWER
W20260924 04:08:29.386427 2613 tablet.cc:2369] T 09195344e10140438ec7945c4f1cb598 P 9c5340a31b954a05aa99bb41395ed290: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 04:08:29.407117 2473 tablet.cc:2369] T 09195344e10140438ec7945c4f1cb598 P b223224d054246d796a20ded0415a012: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 04:08:29.425693 2333 tablet.cc:2369] T 09195344e10140438ec7945c4f1cb598 P 8c9d6c009d9642109c86c0cd26db2f50: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:08:29.477144 2643 raft_consensus.cc:482] T 09195344e10140438ec7945c4f1cb598 P 9c5340a31b954a05aa99bb41395ed290 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:08:29.477315 2643 raft_consensus.cc:504] T 09195344e10140438ec7945c4f1cb598 P 9c5340a31b954a05aa99bb41395ed290 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c9d6c009d9642109c86c0cd26db2f50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46481 } } peers { permanent_uuid: "b223224d054246d796a20ded0415a012" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34761 } } peers { permanent_uuid: "9c5340a31b954a05aa99bb41395ed290" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34579 } }
I20260924 04:08:29.477658 2643 leader_election.cc:288] T 09195344e10140438ec7945c4f1cb598 P 9c5340a31b954a05aa99bb41395ed290 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c9d6c009d9642109c86c0cd26db2f50 (127.0.98.1:46481), b223224d054246d796a20ded0415a012 (127.0.98.2:34761)
I20260924 04:08:29.478157 2283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09195344e10140438ec7945c4f1cb598" candidate_uuid: "9c5340a31b954a05aa99bb41395ed290" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c9d6c009d9642109c86c0cd26db2f50" is_pre_election: true
I20260924 04:08:29.478312 2283 raft_consensus.cc:2454] T 09195344e10140438ec7945c4f1cb598 P 8c9d6c009d9642109c86c0cd26db2f50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c5340a31b954a05aa99bb41395ed290 in term 0.
I20260924 04:08:29.478243 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09195344e10140438ec7945c4f1cb598" candidate_uuid: "9c5340a31b954a05aa99bb41395ed290" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b223224d054246d796a20ded0415a012" is_pre_election: true
I20260924 04:08:29.478500 2419 raft_consensus.cc:2454] T 09195344e10140438ec7945c4f1cb598 P b223224d054246d796a20ded0415a012 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c5340a31b954a05aa99bb41395ed290 in term 0.
I20260924 04:08:29.478667 2497 leader_election.cc:302] T 09195344e10140438ec7945c4f1cb598 P 9c5340a31b954a05aa99bb41395ed290 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c9d6c009d9642109c86c0cd26db2f50, 9c5340a31b954a05aa99bb41395ed290; no voters:
I20260924 04:08:29.478884 2643 raft_consensus.cc:2781] T 09195344e10140438ec7945c4f1cb598 P 9c5340a31b954a05aa99bb41395ed290 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:08:29.478948 2643 raft_consensus.cc:482] T 09195344e10140438ec7945c4f1cb598 P 9c5340a31b954a05aa99bb41395ed290 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:08:29.478999 2643 raft_consensus.cc:3037] T 09195344e10140438ec7945c4f1cb598 P 9c5340a31b954a05aa99bb41395ed290 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:08:29.480041 2643 raft_consensus.cc:504] T 09195344e10140438ec7945c4f1cb598 P 9c5340a31b954a05aa99bb41395ed290 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c9d6c009d9642109c86c0cd26db2f50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46481 } } peers { permanent_uuid: "b223224d054246d796a20ded0415a012" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34761 } } peers { permanent_uuid: "9c5340a31b954a05aa99bb41395ed290" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34579 } }
I20260924 04:08:29.480352 2643 leader_election.cc:288] T 09195344e10140438ec7945c4f1cb598 P 9c5340a31b954a05aa99bb41395ed290 [CANDIDATE]: Term 1 election: Requested vote from peers 8c9d6c009d9642109c86c0cd26db2f50 (127.0.98.1:46481), b223224d054246d796a20ded0415a012 (127.0.98.2:34761)
I20260924 04:08:29.480667 2283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09195344e10140438ec7945c4f1cb598" candidate_uuid: "9c5340a31b954a05aa99bb41395ed290" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c9d6c009d9642109c86c0cd26db2f50"
I20260924 04:08:29.480768 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09195344e10140438ec7945c4f1cb598" candidate_uuid: "9c5340a31b954a05aa99bb41395ed290" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b223224d054246d796a20ded0415a012"
I20260924 04:08:29.480842 2283 raft_consensus.cc:3037] T 09195344e10140438ec7945c4f1cb598 P 8c9d6c009d9642109c86c0cd26db2f50 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:08:29.480911 2419 raft_consensus.cc:3037] T 09195344e10140438ec7945c4f1cb598 P b223224d054246d796a20ded0415a012 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:08:29.481781 2283 raft_consensus.cc:2454] T 09195344e10140438ec7945c4f1cb598 P 8c9d6c009d9642109c86c0cd26db2f50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c5340a31b954a05aa99bb41395ed290 in term 1.
I20260924 04:08:29.481909 2419 raft_consensus.cc:2454] T 09195344e10140438ec7945c4f1cb598 P b223224d054246d796a20ded0415a012 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c5340a31b954a05aa99bb41395ed290 in term 1.
I20260924 04:08:29.482069 2497 leader_election.cc:302] T 09195344e10140438ec7945c4f1cb598 P 9c5340a31b954a05aa99bb41395ed290 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c9d6c009d9642109c86c0cd26db2f50, 9c5340a31b954a05aa99bb41395ed290; no voters:
I20260924 04:08:29.482352 2643 raft_consensus.cc:2781] T 09195344e10140438ec7945c4f1cb598 P 9c5340a31b954a05aa99bb41395ed290 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:08:29.482425 2643 raft_consensus.cc:686] T 09195344e10140438ec7945c4f1cb598 P 9c5340a31b954a05aa99bb41395ed290 [term 1 LEADER]: Becoming Leader. State: Replica: 9c5340a31b954a05aa99bb41395ed290, State: Running, Role: LEADER
I20260924 04:08:29.482589 2643 consensus_queue.cc:237] T 09195344e10140438ec7945c4f1cb598 P 9c5340a31b954a05aa99bb41395ed290 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c9d6c009d9642109c86c0cd26db2f50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46481 } } peers { permanent_uuid: "b223224d054246d796a20ded0415a012" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34761 } } peers { permanent_uuid: "9c5340a31b954a05aa99bb41395ed290" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34579 } }
I20260924 04:08:29.483330 2645 tablet_replica.cc:439] T 09195344e10140438ec7945c4f1cb598 P 9c5340a31b954a05aa99bb41395ed290: TxnStatusTablet state changed. Reason: New leader 9c5340a31b954a05aa99bb41395ed290. Latest consensus state: current_term: 1 leader_uuid: "9c5340a31b954a05aa99bb41395ed290" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c9d6c009d9642109c86c0cd26db2f50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46481 } } peers { permanent_uuid: "b223224d054246d796a20ded0415a012" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34761 } } peers { permanent_uuid: "9c5340a31b954a05aa99bb41395ed290" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34579 } } }
I20260924 04:08:29.483443 2645 tablet_replica.cc:442] T 09195344e10140438ec7945c4f1cb598 P 9c5340a31b954a05aa99bb41395ed290: This TxnStatusTablet replica's current role is: LEADER
I20260924 04:08:29.483752 2657 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 04:08:29.484002 2657 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 04:08:29.484520 2127 catalog_manager.cc:5495] T 09195344e10140438ec7945c4f1cb598 P 9c5340a31b954a05aa99bb41395ed290 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c5340a31b954a05aa99bb41395ed290 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "9c5340a31b954a05aa99bb41395ed290" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c9d6c009d9642109c86c0cd26db2f50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b223224d054246d796a20ded0415a012" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c5340a31b954a05aa99bb41395ed290" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34579 } health_report { overall_health: HEALTHY } } }
I20260924 04:08:29.514792 2652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:29.510735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34579 (local address 127.0.0.1:60714)
0924 04:08:29.510890 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:29.510909 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:29.511009 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:29.511421 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:29.511424 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:29.511434 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 04:08:29.511524 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:29.511528 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:29.512517 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:29.512519 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:29.513166 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:29.513174 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:29.513722 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:29.513725 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:29.513788 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:29.513839 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 04:08:29.514566 (+ 727us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 04:08:29.514576 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 04:08:29.514621 (+ 45us) client_negotiation.cc:240] Negotiation successful
0924 04:08:29.514683 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260924 04:08:29.514839 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:29.510827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60714 (local address 127.0.98.3:34579)
0924 04:08:29.510945 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:29.510950 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:29.510965 (+ 15us) server_negotiation.cc:407] Connection header received
0924 04:08:29.511191 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:29.511194 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:29.511287 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:29.511390 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:08:29.511627 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:29.512409 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:29.513279 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:29.513618 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:29.513740 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:29.513972 (+ 232us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:08:29.514475 (+ 503us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:08:29.514585 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:29.514642 (+ 57us) server_negotiation.cc:299] Negotiation successful
0924 04:08:29.514699 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 04:08:29.519785 2419 raft_consensus.cc:1261] T 09195344e10140438ec7945c4f1cb598 P b223224d054246d796a20ded0415a012 [term 1 FOLLOWER]: Refusing update from remote peer 9c5340a31b954a05aa99bb41395ed290: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:08:29.519934 2283 raft_consensus.cc:1261] T 09195344e10140438ec7945c4f1cb598 P 8c9d6c009d9642109c86c0cd26db2f50 [term 1 FOLLOWER]: Refusing update from remote peer 9c5340a31b954a05aa99bb41395ed290: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:08:29.520547 2645 consensus_queue.cc:1035] T 09195344e10140438ec7945c4f1cb598 P 9c5340a31b954a05aa99bb41395ed290 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b223224d054246d796a20ded0415a012" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:08:29.520753 2643 consensus_queue.cc:1035] T 09195344e10140438ec7945c4f1cb598 P 9c5340a31b954a05aa99bb41395ed290 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c9d6c009d9642109c86c0cd26db2f50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:08:29.524160 2641 tablet_replica.cc:439] T 09195344e10140438ec7945c4f1cb598 P b223224d054246d796a20ded0415a012: TxnStatusTablet state changed. Reason: New leader 9c5340a31b954a05aa99bb41395ed290. Latest consensus state: current_term: 1 leader_uuid: "9c5340a31b954a05aa99bb41395ed290" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c9d6c009d9642109c86c0cd26db2f50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46481 } } peers { permanent_uuid: "b223224d054246d796a20ded0415a012" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34761 } } peers { permanent_uuid: "9c5340a31b954a05aa99bb41395ed290" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34579 } } }
I20260924 04:08:29.524277 2641 tablet_replica.cc:442] T 09195344e10140438ec7945c4f1cb598 P b223224d054246d796a20ded0415a012: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:08:29.524652 2642 tablet_replica.cc:439] T 09195344e10140438ec7945c4f1cb598 P 8c9d6c009d9642109c86c0cd26db2f50: TxnStatusTablet state changed. Reason: New leader 9c5340a31b954a05aa99bb41395ed290. Latest consensus state: current_term: 1 leader_uuid: "9c5340a31b954a05aa99bb41395ed290" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c9d6c009d9642109c86c0cd26db2f50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46481 } } peers { permanent_uuid: "b223224d054246d796a20ded0415a012" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34761 } } peers { permanent_uuid: "9c5340a31b954a05aa99bb41395ed290" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34579 } } }
I20260924 04:08:29.524813 2642 tablet_replica.cc:442] T 09195344e10140438ec7945c4f1cb598 P 8c9d6c009d9642109c86c0cd26db2f50: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:08:29.528869 2643 tablet_replica.cc:439] T 09195344e10140438ec7945c4f1cb598 P 9c5340a31b954a05aa99bb41395ed290: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9c5340a31b954a05aa99bb41395ed290" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c9d6c009d9642109c86c0cd26db2f50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46481 } } peers { permanent_uuid: "b223224d054246d796a20ded0415a012" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34761 } } peers { permanent_uuid: "9c5340a31b954a05aa99bb41395ed290" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34579 } } }
I20260924 04:08:29.528993 2643 tablet_replica.cc:442] T 09195344e10140438ec7945c4f1cb598 P 9c5340a31b954a05aa99bb41395ed290: This TxnStatusTablet replica's current role is: LEADER
I20260924 04:08:29.529520 2643 tablet_replica.cc:439] T 09195344e10140438ec7945c4f1cb598 P 9c5340a31b954a05aa99bb41395ed290: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9c5340a31b954a05aa99bb41395ed290" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c9d6c009d9642109c86c0cd26db2f50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46481 } } peers { permanent_uuid: "b223224d054246d796a20ded0415a012" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34761 } } peers { permanent_uuid: "9c5340a31b954a05aa99bb41395ed290" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34579 } } }
I20260924 04:08:29.529626 2643 tablet_replica.cc:442] T 09195344e10140438ec7945c4f1cb598 P 9c5340a31b954a05aa99bb41395ed290: This TxnStatusTablet replica's current role is: LEADER
I20260924 04:08:29.529681 2641 tablet_replica.cc:439] T 09195344e10140438ec7945c4f1cb598 P b223224d054246d796a20ded0415a012: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9c5340a31b954a05aa99bb41395ed290" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c9d6c009d9642109c86c0cd26db2f50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46481 } } peers { permanent_uuid: "b223224d054246d796a20ded0415a012" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34761 } } peers { permanent_uuid: "9c5340a31b954a05aa99bb41395ed290" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34579 } } }
I20260924 04:08:29.529798 2641 tablet_replica.cc:442] T 09195344e10140438ec7945c4f1cb598 P b223224d054246d796a20ded0415a012: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:08:29.530184 2660 mvcc.cc:205] Tried to move back new op lower bound from 7332753037386645504 to 7332753037242089472. Current Snapshot: MvccSnapshot[applied={T|T < 7332753037386645504}]
I20260924 04:08:29.530435 2662 mvcc.cc:205] Tried to move back new op lower bound from 7332753037386645504 to 7332753037242089472. Current Snapshot: MvccSnapshot[applied={T|T < 7332753037386645504}]
I20260924 04:08:29.532332 2642 tablet_replica.cc:439] T 09195344e10140438ec7945c4f1cb598 P 8c9d6c009d9642109c86c0cd26db2f50: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9c5340a31b954a05aa99bb41395ed290" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c9d6c009d9642109c86c0cd26db2f50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46481 } } peers { permanent_uuid: "b223224d054246d796a20ded0415a012" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34761 } } peers { permanent_uuid: "9c5340a31b954a05aa99bb41395ed290" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34579 } } }
I20260924 04:08:29.532449 2642 tablet_replica.cc:442] T 09195344e10140438ec7945c4f1cb598 P 8c9d6c009d9642109c86c0cd26db2f50: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:08:29.542671 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:29.539322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60716 (local address 127.0.98.3:34579)
0924 04:08:29.539441 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:29.539445 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:29.539509 (+ 64us) server_negotiation.cc:407] Connection header received
0924 04:08:29.539616 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:29.539619 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:29.539664 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:29.539739 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:08:29.540019 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:29.540775 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:29.541627 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:29.541968 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:29.542069 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:29.542264 (+ 195us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:08:29.542341 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:08:29.542410 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:29.542495 (+ 85us) server_negotiation.cc:299] Negotiation successful
0924 04:08:29.542542 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 04:08:29.556309 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:29.551784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35479 (local address 127.0.98.3:34579)
0924 04:08:29.551914 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:29.551926 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:29.552026 (+ 100us) server_negotiation.cc:407] Connection header received
0924 04:08:29.552190 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:29.552193 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:29.552248 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:29.552319 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:08:29.552771 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:29.553616 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:29.555068 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:29.555465 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:29.555557 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:29.555861 (+ 304us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:08:29.555937 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:08:29.555998 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:29.556122 (+ 124us) server_negotiation.cc:299] Negotiation successful
0924 04:08:29.556160 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 04:08:29.556309 2618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:29.551688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34579 (local address 127.0.98.3:35479)
0924 04:08:29.551955 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:29.551972 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:29.552088 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:29.552509 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:29.552514 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:29.552524 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 04:08:29.552659 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:29.552665 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:29.553735 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:29.553739 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:29.554910 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:29.554919 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:29.555565 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:29.555568 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:29.555662 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:29.555707 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 04:08:29.556024 (+ 317us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 04:08:29.556028 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 04:08:29.556102 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 04:08:29.556171 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":5}
I20260924 04:08:29.565323 2419 raft_consensus.cc:1261] T 435cbf6ae4754312bcf1858b6ad9c57b P b223224d054246d796a20ded0415a012 [term 1 FOLLOWER]: Refusing update from remote peer 9c5340a31b954a05aa99bb41395ed290: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:08:29.565656 2283 raft_consensus.cc:1261] T 435cbf6ae4754312bcf1858b6ad9c57b P 8c9d6c009d9642109c86c0cd26db2f50 [term 1 FOLLOWER]: Refusing update from remote peer 9c5340a31b954a05aa99bb41395ed290: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:08:29.566020 2643 consensus_queue.cc:1035] T 435cbf6ae4754312bcf1858b6ad9c57b P 9c5340a31b954a05aa99bb41395ed290 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c9d6c009d9642109c86c0cd26db2f50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:08:29.568423 2643 consensus_queue.cc:1035] T 435cbf6ae4754312bcf1858b6ad9c57b P 9c5340a31b954a05aa99bb41395ed290 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b223224d054246d796a20ded0415a012" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:08:29.572229 2661 mvcc.cc:205] Tried to move back new op lower bound from 7332753037575450624 to 7332753036431060992. Current Snapshot: MvccSnapshot[applied={T|T < 7332753037575450624}]
I20260924 04:08:29.618608 2128 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:08:29.618793 2128 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:08:29.621237 2128 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e422a7c9985241bf963fe50b6dedba65: Sending DeleteTablet for 3 replicas of tablet 435cbf6ae4754312bcf1858b6ad9c57b
I20260924 04:08:29.621805 2546 tablet_service.cc:1514] Processing DeleteTablet for tablet 435cbf6ae4754312bcf1858b6ad9c57b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:08:29 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47332
I20260924 04:08:29.622309 2406 tablet_service.cc:1514] Processing DeleteTablet for tablet 435cbf6ae4754312bcf1858b6ad9c57b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:08:29 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56744
I20260924 04:08:29.622313 2266 tablet_service.cc:1514] Processing DeleteTablet for tablet 435cbf6ae4754312bcf1858b6ad9c57b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:08:29 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53524
I20260924 04:08:29.622802 2677 tablet_replica.cc:330] T 435cbf6ae4754312bcf1858b6ad9c57b P 8c9d6c009d9642109c86c0cd26db2f50: stopping tablet replica
I20260924 04:08:29.622866 2676 tablet_replica.cc:330] T 435cbf6ae4754312bcf1858b6ad9c57b P 9c5340a31b954a05aa99bb41395ed290: stopping tablet replica
I20260924 04:08:29.623106 2676 raft_consensus.cc:2229] T 435cbf6ae4754312bcf1858b6ad9c57b P 9c5340a31b954a05aa99bb41395ed290 [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:08:29.623106 2677 raft_consensus.cc:2229] T 435cbf6ae4754312bcf1858b6ad9c57b P 8c9d6c009d9642109c86c0cd26db2f50 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:08:29.623299 2677 raft_consensus.cc:2258] T 435cbf6ae4754312bcf1858b6ad9c57b P 8c9d6c009d9642109c86c0cd26db2f50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:08:29.623440 2676 raft_consensus.cc:2258] T 435cbf6ae4754312bcf1858b6ad9c57b P 9c5340a31b954a05aa99bb41395ed290 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:08:29.623660 2678 tablet_replica.cc:330] T 435cbf6ae4754312bcf1858b6ad9c57b P b223224d054246d796a20ded0415a012: stopping tablet replica
I20260924 04:08:29.623610 2128 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53412:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 04:08:29.623970 2678 raft_consensus.cc:2229] T 435cbf6ae4754312bcf1858b6ad9c57b P b223224d054246d796a20ded0415a012 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260924 04:08:29.624019 2128 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:08:29.624143 2678 raft_consensus.cc:2258] T 435cbf6ae4754312bcf1858b6ad9c57b P b223224d054246d796a20ded0415a012 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:08:29.624853 2677 ts_tablet_manager.cc:1905] T 435cbf6ae4754312bcf1858b6ad9c57b P 8c9d6c009d9642109c86c0cd26db2f50: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:08:29.625844 2676 ts_tablet_manager.cc:1905] T 435cbf6ae4754312bcf1858b6ad9c57b P 9c5340a31b954a05aa99bb41395ed290: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:08:29.626012 2678 ts_tablet_manager.cc:1905] T 435cbf6ae4754312bcf1858b6ad9c57b P b223224d054246d796a20ded0415a012: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:08:29.627915 2677 ts_tablet_manager.cc:1918] T 435cbf6ae4754312bcf1858b6ad9c57b P 8c9d6c009d9642109c86c0cd26db2f50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 04:08:29.628005 2677 log.cc:1196] T 435cbf6ae4754312bcf1858b6ad9c57b P 8c9d6c009d9642109c86c0cd26db2f50: Deleting WAL directory at /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/minicluster-data/ts-0/wal/wals/435cbf6ae4754312bcf1858b6ad9c57b
I20260924 04:08:29.628333 2677 ts_tablet_manager.cc:1939] T 435cbf6ae4754312bcf1858b6ad9c57b P 8c9d6c009d9642109c86c0cd26db2f50: Deleting consensus metadata
I20260924 04:08:29.628539 2676 ts_tablet_manager.cc:1918] T 435cbf6ae4754312bcf1858b6ad9c57b P 9c5340a31b954a05aa99bb41395ed290: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 04:08:29.628540 2678 ts_tablet_manager.cc:1918] T 435cbf6ae4754312bcf1858b6ad9c57b P b223224d054246d796a20ded0415a012: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 04:08:29.628628 2676 log.cc:1196] T 435cbf6ae4754312bcf1858b6ad9c57b P 9c5340a31b954a05aa99bb41395ed290: Deleting WAL directory at /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/minicluster-data/ts-2/wal/wals/435cbf6ae4754312bcf1858b6ad9c57b
I20260924 04:08:29.628628 2678 log.cc:1196] T 435cbf6ae4754312bcf1858b6ad9c57b P b223224d054246d796a20ded0415a012: Deleting WAL directory at /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/minicluster-data/ts-1/wal/wals/435cbf6ae4754312bcf1858b6ad9c57b
I20260924 04:08:29.628962 2678 ts_tablet_manager.cc:1939] T 435cbf6ae4754312bcf1858b6ad9c57b P b223224d054246d796a20ded0415a012: Deleting consensus metadata
I20260924 04:08:29.628998 2676 ts_tablet_manager.cc:1939] T 435cbf6ae4754312bcf1858b6ad9c57b P 9c5340a31b954a05aa99bb41395ed290: Deleting consensus metadata
I20260924 04:08:29.629644 2546 tablet_service.cc:1467] Processing CreateTablet for tablet 8107b8076e2f4e9d8e233961e9c3edb3 (DEFAULT_TABLE table=test-table [id=05ecb349af914a2da60d034159fb4977]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:08:29.629889 2546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8107b8076e2f4e9d8e233961e9c3edb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:08:29.629906 2266 tablet_service.cc:1467] Processing CreateTablet for tablet 8107b8076e2f4e9d8e233961e9c3edb3 (DEFAULT_TABLE table=test-table [id=05ecb349af914a2da60d034159fb4977]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:08:29.629961 2115 catalog_manager.cc:4841] TS 9c5340a31b954a05aa99bb41395ed290 (127.0.98.3:34579): tablet 435cbf6ae4754312bcf1858b6ad9c57b (table test-table [id=a7faa8c8be254badb078a9509a1f587a]) successfully deleted
I20260924 04:08:29.630157 2266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8107b8076e2f4e9d8e233961e9c3edb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:08:29.631639 2637 tablet_bootstrap.cc:492] T 8107b8076e2f4e9d8e233961e9c3edb3 P 9c5340a31b954a05aa99bb41395ed290: Bootstrap starting.
I20260924 04:08:29.632360 2406 tablet_service.cc:1467] Processing CreateTablet for tablet 8107b8076e2f4e9d8e233961e9c3edb3 (DEFAULT_TABLE table=test-table [id=05ecb349af914a2da60d034159fb4977]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:08:29.632525 2637 tablet_bootstrap.cc:654] T 8107b8076e2f4e9d8e233961e9c3edb3 P 9c5340a31b954a05aa99bb41395ed290: Neither blocks nor log segments found. Creating new log.
I20260924 04:08:29.632597 2406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8107b8076e2f4e9d8e233961e9c3edb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:08:29.632810 2635 tablet_bootstrap.cc:492] T 8107b8076e2f4e9d8e233961e9c3edb3 P 8c9d6c009d9642109c86c0cd26db2f50: Bootstrap starting.
I20260924 04:08:29.633611 2637 tablet_bootstrap.cc:492] T 8107b8076e2f4e9d8e233961e9c3edb3 P 9c5340a31b954a05aa99bb41395ed290: No bootstrap required, opened a new log
I20260924 04:08:29.633665 2637 ts_tablet_manager.cc:1397] T 8107b8076e2f4e9d8e233961e9c3edb3 P 9c5340a31b954a05aa99bb41395ed290: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 04:08:29.633671 2635 tablet_bootstrap.cc:654] T 8107b8076e2f4e9d8e233961e9c3edb3 P 8c9d6c009d9642109c86c0cd26db2f50: Neither blocks nor log segments found. Creating new log.
I20260924 04:08:29.633978 2637 raft_consensus.cc:348] T 8107b8076e2f4e9d8e233961e9c3edb3 P 9c5340a31b954a05aa99bb41395ed290 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c9d6c009d9642109c86c0cd26db2f50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46481 } } peers { permanent_uuid: "b223224d054246d796a20ded0415a012" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34761 } } peers { permanent_uuid: "9c5340a31b954a05aa99bb41395ed290" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34579 } }
I20260924 04:08:29.634057 2637 raft_consensus.cc:374] T 8107b8076e2f4e9d8e233961e9c3edb3 P 9c5340a31b954a05aa99bb41395ed290 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:08:29.634078 2637 raft_consensus.cc:729] T 8107b8076e2f4e9d8e233961e9c3edb3 P 9c5340a31b954a05aa99bb41395ed290 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c5340a31b954a05aa99bb41395ed290, State: Initialized, Role: FOLLOWER
I20260924 04:08:29.634203 2637 consensus_queue.cc:260] T 8107b8076e2f4e9d8e233961e9c3edb3 P 9c5340a31b954a05aa99bb41395ed290 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c9d6c009d9642109c86c0cd26db2f50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46481 } } peers { permanent_uuid: "b223224d054246d796a20ded0415a012" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34761 } } peers { permanent_uuid: "9c5340a31b954a05aa99bb41395ed290" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34579 } }
I20260924 04:08:29.634331 2113 catalog_manager.cc:4841] TS b223224d054246d796a20ded0415a012 (127.0.98.2:34761): tablet 435cbf6ae4754312bcf1858b6ad9c57b (table test-table [id=a7faa8c8be254badb078a9509a1f587a]) successfully deleted
I20260924 04:08:29.634476 2113 catalog_manager.cc:4841] TS 8c9d6c009d9642109c86c0cd26db2f50 (127.0.98.1:46481): tablet 435cbf6ae4754312bcf1858b6ad9c57b (table test-table [id=a7faa8c8be254badb078a9509a1f587a]) successfully deleted
I20260924 04:08:29.634505 2637 ts_tablet_manager.cc:1428] T 8107b8076e2f4e9d8e233961e9c3edb3 P 9c5340a31b954a05aa99bb41395ed290: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:08:29.637037 2636 tablet_bootstrap.cc:492] T 8107b8076e2f4e9d8e233961e9c3edb3 P b223224d054246d796a20ded0415a012: Bootstrap starting.
I20260924 04:08:29.638000 2636 tablet_bootstrap.cc:654] T 8107b8076e2f4e9d8e233961e9c3edb3 P b223224d054246d796a20ded0415a012: Neither blocks nor log segments found. Creating new log.
I20260924 04:08:29.638501 2635 tablet_bootstrap.cc:492] T 8107b8076e2f4e9d8e233961e9c3edb3 P 8c9d6c009d9642109c86c0cd26db2f50: No bootstrap required, opened a new log
I20260924 04:08:29.638593 2635 ts_tablet_manager.cc:1397] T 8107b8076e2f4e9d8e233961e9c3edb3 P 8c9d6c009d9642109c86c0cd26db2f50: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260924 04:08:29.639063 2635 raft_consensus.cc:348] T 8107b8076e2f4e9d8e233961e9c3edb3 P 8c9d6c009d9642109c86c0cd26db2f50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c9d6c009d9642109c86c0cd26db2f50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46481 } } peers { permanent_uuid: "b223224d054246d796a20ded0415a012" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34761 } } peers { permanent_uuid: "9c5340a31b954a05aa99bb41395ed290" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34579 } }
I20260924 04:08:29.639200 2635 raft_consensus.cc:374] T 8107b8076e2f4e9d8e233961e9c3edb3 P 8c9d6c009d9642109c86c0cd26db2f50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:08:29.639245 2635 raft_consensus.cc:729] T 8107b8076e2f4e9d8e233961e9c3edb3 P 8c9d6c009d9642109c86c0cd26db2f50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c9d6c009d9642109c86c0cd26db2f50, State: Initialized, Role: FOLLOWER
I20260924 04:08:29.639365 2635 consensus_queue.cc:260] T 8107b8076e2f4e9d8e233961e9c3edb3 P 8c9d6c009d9642109c86c0cd26db2f50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c9d6c009d9642109c86c0cd26db2f50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46481 } } peers { permanent_uuid: "b223224d054246d796a20ded0415a012" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34761 } } peers { permanent_uuid: "9c5340a31b954a05aa99bb41395ed290" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34579 } }
I20260924 04:08:29.639634 2635 ts_tablet_manager.cc:1428] T 8107b8076e2f4e9d8e233961e9c3edb3 P 8c9d6c009d9642109c86c0cd26db2f50: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:08:29.640540 2636 tablet_bootstrap.cc:492] T 8107b8076e2f4e9d8e233961e9c3edb3 P b223224d054246d796a20ded0415a012: No bootstrap required, opened a new log
I20260924 04:08:29.640638 2636 ts_tablet_manager.cc:1397] T 8107b8076e2f4e9d8e233961e9c3edb3 P b223224d054246d796a20ded0415a012: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260924 04:08:29.641158 2636 raft_consensus.cc:348] T 8107b8076e2f4e9d8e233961e9c3edb3 P b223224d054246d796a20ded0415a012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c9d6c009d9642109c86c0cd26db2f50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46481 } } peers { permanent_uuid: "b223224d054246d796a20ded0415a012" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34761 } } peers { permanent_uuid: "9c5340a31b954a05aa99bb41395ed290" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34579 } }
I20260924 04:08:29.641280 2636 raft_consensus.cc:374] T 8107b8076e2f4e9d8e233961e9c3edb3 P b223224d054246d796a20ded0415a012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:08:29.641319 2636 raft_consensus.cc:729] T 8107b8076e2f4e9d8e233961e9c3edb3 P b223224d054246d796a20ded0415a012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b223224d054246d796a20ded0415a012, State: Initialized, Role: FOLLOWER
I20260924 04:08:29.641458 2636 consensus_queue.cc:260] T 8107b8076e2f4e9d8e233961e9c3edb3 P b223224d054246d796a20ded0415a012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c9d6c009d9642109c86c0cd26db2f50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46481 } } peers { permanent_uuid: "b223224d054246d796a20ded0415a012" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34761 } } peers { permanent_uuid: "9c5340a31b954a05aa99bb41395ed290" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34579 } }
I20260924 04:08:29.643074 2636 ts_tablet_manager.cc:1428] T 8107b8076e2f4e9d8e233961e9c3edb3 P b223224d054246d796a20ded0415a012: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260924 04:08:29.858248 2642 raft_consensus.cc:482] T 8107b8076e2f4e9d8e233961e9c3edb3 P 8c9d6c009d9642109c86c0cd26db2f50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:08:29.858398 2642 raft_consensus.cc:504] T 8107b8076e2f4e9d8e233961e9c3edb3 P 8c9d6c009d9642109c86c0cd26db2f50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c9d6c009d9642109c86c0cd26db2f50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46481 } } peers { permanent_uuid: "b223224d054246d796a20ded0415a012" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34761 } } peers { permanent_uuid: "9c5340a31b954a05aa99bb41395ed290" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34579 } }
I20260924 04:08:29.859527 2642 leader_election.cc:288] T 8107b8076e2f4e9d8e233961e9c3edb3 P 8c9d6c009d9642109c86c0cd26db2f50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b223224d054246d796a20ded0415a012 (127.0.98.2:34761), 9c5340a31b954a05aa99bb41395ed290 (127.0.98.3:34579)
I20260924 04:08:29.863998 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:29.859864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47643 (local address 127.0.98.3:34579)
0924 04:08:29.859995 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:29.859999 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:29.860190 (+ 191us) server_negotiation.cc:407] Connection header received
0924 04:08:29.860271 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:29.860274 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:29.860332 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:29.860414 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:08:29.860799 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:29.861627 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:29.862978 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:29.863464 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:29.863578 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:29.863641 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:29.863833 (+ 192us) server_negotiation.cc:299] Negotiation successful
0924 04:08:29.863903 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 04:08:29.864041 2684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:29.859815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34579 (local address 127.0.98.1:47643)
0924 04:08:29.860091 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:29.860106 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:29.860180 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:29.860465 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:29.860468 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:29.860479 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:08:29.860667 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:29.860694 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:29.861722 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:29.861726 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:29.862849 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:29.862860 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:29.863593 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:29.863597 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:29.863696 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:29.863718 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 04:08:29.863796 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 04:08:29.863876 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"mutex_wait_us":190,"spinlock_wait_cycles":896,"thread_start_us":78,"threads_started":1}
I20260924 04:08:29.864715 2566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8107b8076e2f4e9d8e233961e9c3edb3" candidate_uuid: "8c9d6c009d9642109c86c0cd26db2f50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c5340a31b954a05aa99bb41395ed290" is_pre_election: true
I20260924 04:08:29.864946 2683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:29.859810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45695 (local address 127.0.98.2:34761)
0924 04:08:29.860140 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:29.860173 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:29.860189 (+ 16us) server_negotiation.cc:407] Connection header received
0924 04:08:29.860271 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:29.860274 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:29.860326 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:29.860416 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:08:29.860806 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:29.861596 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:29.863711 (+ 2115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:29.864367 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:29.864474 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:29.864545 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:29.864733 (+ 188us) server_negotiation.cc:299] Negotiation successful
0924 04:08:29.864811 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":55,"threads_started":1}
I20260924 04:08:29.864950 2682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:29.859722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34761 (local address 127.0.98.1:45695)
0924 04:08:29.860041 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:29.860057 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:29.860180 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:29.860447 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:29.860450 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:29.860470 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:08:29.860694 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:29.860700 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:29.861722 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:29.861726 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:29.862842 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:29.862859 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:29.864508 (+ 1649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:29.864511 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:29.864577 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:29.864595 (+ 18us) client_negotiation.cc:769] Sending connection context
0924 04:08:29.864667 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 04:08:29.864789 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"mutex_wait_us":145,"spinlock_wait_cycles":1280,"thread_start_us":112,"threads_started":1}
I20260924 04:08:29.864984 2566 raft_consensus.cc:2454] T 8107b8076e2f4e9d8e233961e9c3edb3 P 9c5340a31b954a05aa99bb41395ed290 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c9d6c009d9642109c86c0cd26db2f50 in term 0.
I20260924 04:08:29.865468 2219 leader_election.cc:302] T 8107b8076e2f4e9d8e233961e9c3edb3 P 8c9d6c009d9642109c86c0cd26db2f50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c9d6c009d9642109c86c0cd26db2f50, 9c5340a31b954a05aa99bb41395ed290; no voters:
I20260924 04:08:29.865561 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8107b8076e2f4e9d8e233961e9c3edb3" candidate_uuid: "8c9d6c009d9642109c86c0cd26db2f50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b223224d054246d796a20ded0415a012" is_pre_election: true
I20260924 04:08:29.865738 2642 raft_consensus.cc:2781] T 8107b8076e2f4e9d8e233961e9c3edb3 P 8c9d6c009d9642109c86c0cd26db2f50 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:08:29.865747 2419 raft_consensus.cc:2454] T 8107b8076e2f4e9d8e233961e9c3edb3 P b223224d054246d796a20ded0415a012 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c9d6c009d9642109c86c0cd26db2f50 in term 0.
I20260924 04:08:29.865795 2642 raft_consensus.cc:482] T 8107b8076e2f4e9d8e233961e9c3edb3 P 8c9d6c009d9642109c86c0cd26db2f50 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:08:29.865862 2642 raft_consensus.cc:3037] T 8107b8076e2f4e9d8e233961e9c3edb3 P 8c9d6c009d9642109c86c0cd26db2f50 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:08:29.866940 2642 raft_consensus.cc:504] T 8107b8076e2f4e9d8e233961e9c3edb3 P 8c9d6c009d9642109c86c0cd26db2f50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c9d6c009d9642109c86c0cd26db2f50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46481 } } peers { permanent_uuid: "b223224d054246d796a20ded0415a012" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34761 } } peers { permanent_uuid: "9c5340a31b954a05aa99bb41395ed290" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34579 } }
I20260924 04:08:29.867326 2642 leader_election.cc:288] T 8107b8076e2f4e9d8e233961e9c3edb3 P 8c9d6c009d9642109c86c0cd26db2f50 [CANDIDATE]: Term 1 election: Requested vote from peers b223224d054246d796a20ded0415a012 (127.0.98.2:34761), 9c5340a31b954a05aa99bb41395ed290 (127.0.98.3:34579)
I20260924 04:08:29.867667 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8107b8076e2f4e9d8e233961e9c3edb3" candidate_uuid: "8c9d6c009d9642109c86c0cd26db2f50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b223224d054246d796a20ded0415a012"
I20260924 04:08:29.867671 2566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8107b8076e2f4e9d8e233961e9c3edb3" candidate_uuid: "8c9d6c009d9642109c86c0cd26db2f50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c5340a31b954a05aa99bb41395ed290"
I20260924 04:08:29.867803 2419 raft_consensus.cc:3037] T 8107b8076e2f4e9d8e233961e9c3edb3 P b223224d054246d796a20ded0415a012 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:08:29.867815 2566 raft_consensus.cc:3037] T 8107b8076e2f4e9d8e233961e9c3edb3 P 9c5340a31b954a05aa99bb41395ed290 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:08:29.868813 2566 raft_consensus.cc:2454] T 8107b8076e2f4e9d8e233961e9c3edb3 P 9c5340a31b954a05aa99bb41395ed290 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c9d6c009d9642109c86c0cd26db2f50 in term 1.
I20260924 04:08:29.868821 2419 raft_consensus.cc:2454] T 8107b8076e2f4e9d8e233961e9c3edb3 P b223224d054246d796a20ded0415a012 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c9d6c009d9642109c86c0cd26db2f50 in term 1.
I20260924 04:08:29.869086 2219 leader_election.cc:302] T 8107b8076e2f4e9d8e233961e9c3edb3 P 8c9d6c009d9642109c86c0cd26db2f50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 8c9d6c009d9642109c86c0cd26db2f50, 9c5340a31b954a05aa99bb41395ed290, b223224d054246d796a20ded0415a012; no voters:
I20260924 04:08:29.869230 2642 raft_consensus.cc:2781] T 8107b8076e2f4e9d8e233961e9c3edb3 P 8c9d6c009d9642109c86c0cd26db2f50 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:08:29.869397 2642 raft_consensus.cc:686] T 8107b8076e2f4e9d8e233961e9c3edb3 P 8c9d6c009d9642109c86c0cd26db2f50 [term 1 LEADER]: Becoming Leader. State: Replica: 8c9d6c009d9642109c86c0cd26db2f50, State: Running, Role: LEADER
I20260924 04:08:29.869656 2642 consensus_queue.cc:237] T 8107b8076e2f4e9d8e233961e9c3edb3 P 8c9d6c009d9642109c86c0cd26db2f50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c9d6c009d9642109c86c0cd26db2f50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46481 } } peers { permanent_uuid: "b223224d054246d796a20ded0415a012" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34761 } } peers { permanent_uuid: "9c5340a31b954a05aa99bb41395ed290" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34579 } }
I20260924 04:08:29.872020 2128 catalog_manager.cc:5495] T 8107b8076e2f4e9d8e233961e9c3edb3 P 8c9d6c009d9642109c86c0cd26db2f50 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8c9d6c009d9642109c86c0cd26db2f50 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "8c9d6c009d9642109c86c0cd26db2f50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c9d6c009d9642109c86c0cd26db2f50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b223224d054246d796a20ded0415a012" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c5340a31b954a05aa99bb41395ed290" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34579 } health_report { overall_health: UNKNOWN } } }
I20260924 04:08:29.886878 2687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:29.882939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58180 (local address 127.0.98.1:46481)
0924 04:08:29.883228 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:29.883235 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:29.883246 (+ 11us) server_negotiation.cc:407] Connection header received
0924 04:08:29.883308 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:29.883310 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:29.883357 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:29.883454 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:08:29.883637 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:29.884491 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:29.885502 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:29.885745 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:29.885844 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:29.886028 (+ 184us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:08:29.886473 (+ 445us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:08:29.886563 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:29.886625 (+ 62us) server_negotiation.cc:299] Negotiation successful
0924 04:08:29.886669 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":150,"threads_started":1}
I20260924 04:08:29.896327 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:29.893310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43199 (local address 127.0.98.3:34579)
0924 04:08:29.893433 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:29.893437 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:29.893611 (+ 174us) server_negotiation.cc:407] Connection header received
0924 04:08:29.893713 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:29.893715 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:29.893756 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:29.893820 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:08:29.894030 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:29.894522 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:29.895374 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:29.895585 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:29.895665 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:29.895964 (+ 299us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:08:29.896029 (+ 65us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:08:29.896082 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:29.896181 (+ 99us) server_negotiation.cc:299] Negotiation successful
0924 04:08:29.896221 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 04:08:29.896341 2690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:29.893223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34579 (local address 127.0.98.1:43199)
0924 04:08:29.893537 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:29.893549 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:29.893633 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:29.893847 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:29.893849 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:29.893855 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 04:08:29.893945 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:29.893949 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:29.894615 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:29.894617 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:29.895241 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:29.895249 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:29.895724 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:29.895727 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:29.895803 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:29.895843 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 04:08:29.896103 (+ 260us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 04:08:29.896108 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 04:08:29.896157 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 04:08:29.896222 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":134,"threads_started":1}
I20260924 04:08:29.902343 2419 raft_consensus.cc:1261] T 8107b8076e2f4e9d8e233961e9c3edb3 P b223224d054246d796a20ded0415a012 [term 1 FOLLOWER]: Refusing update from remote peer 8c9d6c009d9642109c86c0cd26db2f50: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:08:29.902501 2566 raft_consensus.cc:1261] T 8107b8076e2f4e9d8e233961e9c3edb3 P 9c5340a31b954a05aa99bb41395ed290 [term 1 FOLLOWER]: Refusing update from remote peer 8c9d6c009d9642109c86c0cd26db2f50: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:08:29.902855 2685 consensus_queue.cc:1035] T 8107b8076e2f4e9d8e233961e9c3edb3 P 8c9d6c009d9642109c86c0cd26db2f50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b223224d054246d796a20ded0415a012" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:08:29.903013 2642 consensus_queue.cc:1035] T 8107b8076e2f4e9d8e233961e9c3edb3 P 8c9d6c009d9642109c86c0cd26db2f50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c5340a31b954a05aa99bb41395ed290" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:08:29.920089 2618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:29.916852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46481 (local address 127.0.98.3:46291)
0924 04:08:29.917004 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:29.917020 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:29.917121 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:29.917371 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:29.917374 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:29.917381 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 04:08:29.917472 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:29.917477 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:29.918213 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:29.918219 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:29.918802 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:29.918809 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:29.919364 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:29.919367 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:29.919459 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:29.919501 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 04:08:29.919815 (+ 314us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 04:08:29.919823 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 04:08:29.919885 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 04:08:29.919964 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260924 04:08:29.920410 2687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:29.916926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46291 (local address 127.0.98.1:46481)
0924 04:08:29.917023 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:29.917029 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:29.917063 (+ 34us) server_negotiation.cc:407] Connection header received
0924 04:08:29.917240 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:29.917244 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:29.917296 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:29.917363 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:08:29.917589 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:29.918097 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:29.918918 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:29.919247 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:29.919350 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:29.919619 (+ 269us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:08:29.919700 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:08:29.919811 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:29.920241 (+ 430us) server_negotiation.cc:299] Negotiation successful
0924 04:08:29.920305 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
W20260924 04:08:29.921054 2661 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 04:08:31.475419 2128 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:08:31.475597 2128 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:08:31.477916 2128 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e422a7c9985241bf963fe50b6dedba65: Sending DeleteTablet for 3 replicas of tablet 8107b8076e2f4e9d8e233961e9c3edb3
I20260924 04:08:31.478562 2266 tablet_service.cc:1514] Processing DeleteTablet for tablet 8107b8076e2f4e9d8e233961e9c3edb3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:08:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53524
I20260924 04:08:31.478574 2545 tablet_service.cc:1514] Processing DeleteTablet for tablet 8107b8076e2f4e9d8e233961e9c3edb3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:08:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47332
I20260924 04:08:31.478602 2406 tablet_service.cc:1514] Processing DeleteTablet for tablet 8107b8076e2f4e9d8e233961e9c3edb3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:08:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56744
I20260924 04:08:31.478931 2715 tablet_replica.cc:330] T 8107b8076e2f4e9d8e233961e9c3edb3 P 8c9d6c009d9642109c86c0cd26db2f50: stopping tablet replica
I20260924 04:08:31.478982 2716 tablet_replica.cc:330] T 8107b8076e2f4e9d8e233961e9c3edb3 P b223224d054246d796a20ded0415a012: stopping tablet replica
I20260924 04:08:31.479126 2716 raft_consensus.cc:2229] T 8107b8076e2f4e9d8e233961e9c3edb3 P b223224d054246d796a20ded0415a012 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:08:31.479116 2715 raft_consensus.cc:2229] T 8107b8076e2f4e9d8e233961e9c3edb3 P 8c9d6c009d9642109c86c0cd26db2f50 [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:08:31.479281 2716 raft_consensus.cc:2258] T 8107b8076e2f4e9d8e233961e9c3edb3 P b223224d054246d796a20ded0415a012 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:08:31.479465 2715 raft_consensus.cc:2258] T 8107b8076e2f4e9d8e233961e9c3edb3 P 8c9d6c009d9642109c86c0cd26db2f50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:08:31.480367 2716 ts_tablet_manager.cc:1905] T 8107b8076e2f4e9d8e233961e9c3edb3 P b223224d054246d796a20ded0415a012: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:08:31.481006 2715 ts_tablet_manager.cc:1905] T 8107b8076e2f4e9d8e233961e9c3edb3 P 8c9d6c009d9642109c86c0cd26db2f50: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:08:31.482034 2717 tablet_replica.cc:330] T 8107b8076e2f4e9d8e233961e9c3edb3 P 9c5340a31b954a05aa99bb41395ed290: stopping tablet replica
I20260924 04:08:31.482194 2717 raft_consensus.cc:2229] T 8107b8076e2f4e9d8e233961e9c3edb3 P 9c5340a31b954a05aa99bb41395ed290 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:08:31.482337 2717 raft_consensus.cc:2258] T 8107b8076e2f4e9d8e233961e9c3edb3 P 9c5340a31b954a05aa99bb41395ed290 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:08:31.482928 2717 ts_tablet_manager.cc:1905] T 8107b8076e2f4e9d8e233961e9c3edb3 P 9c5340a31b954a05aa99bb41395ed290: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:08:31.483045 2716 ts_tablet_manager.cc:1918] T 8107b8076e2f4e9d8e233961e9c3edb3 P b223224d054246d796a20ded0415a012: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 04:08:31.483122 2716 log.cc:1196] T 8107b8076e2f4e9d8e233961e9c3edb3 P b223224d054246d796a20ded0415a012: Deleting WAL directory at /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/minicluster-data/ts-1/wal/wals/8107b8076e2f4e9d8e233961e9c3edb3
I20260924 04:08:31.483443 2716 ts_tablet_manager.cc:1939] T 8107b8076e2f4e9d8e233961e9c3edb3 P b223224d054246d796a20ded0415a012: Deleting consensus metadata
I20260924 04:08:31.483821 2715 ts_tablet_manager.cc:1918] T 8107b8076e2f4e9d8e233961e9c3edb3 P 8c9d6c009d9642109c86c0cd26db2f50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 04:08:31.483901 2715 log.cc:1196] T 8107b8076e2f4e9d8e233961e9c3edb3 P 8c9d6c009d9642109c86c0cd26db2f50: Deleting WAL directory at /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/minicluster-data/ts-0/wal/wals/8107b8076e2f4e9d8e233961e9c3edb3
I20260924 04:08:31.484122 2113 catalog_manager.cc:4841] TS b223224d054246d796a20ded0415a012 (127.0.98.2:34761): tablet 8107b8076e2f4e9d8e233961e9c3edb3 (table test-table [id=05ecb349af914a2da60d034159fb4977]) successfully deleted
I20260924 04:08:31.484197 2715 ts_tablet_manager.cc:1939] T 8107b8076e2f4e9d8e233961e9c3edb3 P 8c9d6c009d9642109c86c0cd26db2f50: Deleting consensus metadata
I20260924 04:08:31.484972 2113 catalog_manager.cc:4841] TS 8c9d6c009d9642109c86c0cd26db2f50 (127.0.98.1:46481): tablet 8107b8076e2f4e9d8e233961e9c3edb3 (table test-table [id=05ecb349af914a2da60d034159fb4977]) successfully deleted
I20260924 04:08:31.485666 2717 ts_tablet_manager.cc:1918] T 8107b8076e2f4e9d8e233961e9c3edb3 P 9c5340a31b954a05aa99bb41395ed290: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 04:08:31.486167 2717 log.cc:1196] T 8107b8076e2f4e9d8e233961e9c3edb3 P 9c5340a31b954a05aa99bb41395ed290: Deleting WAL directory at /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/minicluster-data/ts-2/wal/wals/8107b8076e2f4e9d8e233961e9c3edb3
I20260924 04:08:31.486407 2717 ts_tablet_manager.cc:1939] T 8107b8076e2f4e9d8e233961e9c3edb3 P 9c5340a31b954a05aa99bb41395ed290: Deleting consensus metadata
I20260924 04:08:31.487051 2115 catalog_manager.cc:4841] TS 9c5340a31b954a05aa99bb41395ed290 (127.0.98.3:34579): tablet 8107b8076e2f4e9d8e233961e9c3edb3 (table test-table [id=05ecb349af914a2da60d034159fb4977]) successfully deleted
Sep 24 04:08:31 dist-test-slave-00ld krb5kdc[2084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222911, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 04:08:31 dist-test-slave-00ld krb5kdc[2084](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222911, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:08:31.515156 2725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:31.498472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55234 (local address 127.0.98.62:35555)
0924 04:08:31.498770 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:31.498777 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:31.498849 (+ 72us) server_negotiation.cc:407] Connection header received
0924 04:08:31.499002 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:31.499004 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:31.499047 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:31.499174 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:31.499957 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:31.500500 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:31.501448 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:31.501677 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:31.501781 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:31.512090 (+ 10309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:31.512093 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:31.512095 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:31.512119 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:31.513522 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:31.513933 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:31.513938 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:31.513940 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:31.514020 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:31.514296 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:31.514300 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:31.514301 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:31.514481 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:31.514616 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:31.514862 (+ 246us) server_negotiation.cc:299] Negotiation successful
0924 04:08:31.514986 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":95,"threads_started":1}
I20260924 04:08:31.517715 2128 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55234:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:08:31.518146 2128 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:08:31.522660 2266 tablet_service.cc:1467] Processing CreateTablet for tablet 3a9f9e4e4d6742318b3bf564cf2436bf (DEFAULT_TABLE table=test-table [id=37d55f5cce3e47d6934c262058aed7f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:08:31.522821 2406 tablet_service.cc:1467] Processing CreateTablet for tablet 3a9f9e4e4d6742318b3bf564cf2436bf (DEFAULT_TABLE table=test-table [id=37d55f5cce3e47d6934c262058aed7f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:08:31.522845 2545 tablet_service.cc:1467] Processing CreateTablet for tablet 3a9f9e4e4d6742318b3bf564cf2436bf (DEFAULT_TABLE table=test-table [id=37d55f5cce3e47d6934c262058aed7f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:08:31.522958 2266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a9f9e4e4d6742318b3bf564cf2436bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:08:31.523097 2406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a9f9e4e4d6742318b3bf564cf2436bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:08:31.523093 2545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a9f9e4e4d6742318b3bf564cf2436bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:08:31.525622 2728 tablet_bootstrap.cc:492] T 3a9f9e4e4d6742318b3bf564cf2436bf P 8c9d6c009d9642109c86c0cd26db2f50: Bootstrap starting.
I20260924 04:08:31.525632 2726 tablet_bootstrap.cc:492] T 3a9f9e4e4d6742318b3bf564cf2436bf P b223224d054246d796a20ded0415a012: Bootstrap starting.
I20260924 04:08:31.526360 2726 tablet_bootstrap.cc:654] T 3a9f9e4e4d6742318b3bf564cf2436bf P b223224d054246d796a20ded0415a012: Neither blocks nor log segments found. Creating new log.
I20260924 04:08:31.526371 2728 tablet_bootstrap.cc:654] T 3a9f9e4e4d6742318b3bf564cf2436bf P 8c9d6c009d9642109c86c0cd26db2f50: Neither blocks nor log segments found. Creating new log.
I20260924 04:08:31.526933 2727 tablet_bootstrap.cc:492] T 3a9f9e4e4d6742318b3bf564cf2436bf P 9c5340a31b954a05aa99bb41395ed290: Bootstrap starting.
I20260924 04:08:31.527380 2726 tablet_bootstrap.cc:492] T 3a9f9e4e4d6742318b3bf564cf2436bf P b223224d054246d796a20ded0415a012: No bootstrap required, opened a new log
I20260924 04:08:31.527449 2726 ts_tablet_manager.cc:1397] T 3a9f9e4e4d6742318b3bf564cf2436bf P b223224d054246d796a20ded0415a012: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 04:08:31.527715 2727 tablet_bootstrap.cc:654] T 3a9f9e4e4d6742318b3bf564cf2436bf P 9c5340a31b954a05aa99bb41395ed290: Neither blocks nor log segments found. Creating new log.
I20260924 04:08:31.527943 2726 raft_consensus.cc:348] T 3a9f9e4e4d6742318b3bf564cf2436bf P b223224d054246d796a20ded0415a012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c9d6c009d9642109c86c0cd26db2f50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46481 } } peers { permanent_uuid: "b223224d054246d796a20ded0415a012" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34761 } } peers { permanent_uuid: "9c5340a31b954a05aa99bb41395ed290" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34579 } }
I20260924 04:08:31.528071 2726 raft_consensus.cc:374] T 3a9f9e4e4d6742318b3bf564cf2436bf P b223224d054246d796a20ded0415a012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:08:31.528131 2726 raft_consensus.cc:729] T 3a9f9e4e4d6742318b3bf564cf2436bf P b223224d054246d796a20ded0415a012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b223224d054246d796a20ded0415a012, State: Initialized, Role: FOLLOWER
I20260924 04:08:31.528301 2726 consensus_queue.cc:260] T 3a9f9e4e4d6742318b3bf564cf2436bf P b223224d054246d796a20ded0415a012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c9d6c009d9642109c86c0cd26db2f50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46481 } } peers { permanent_uuid: "b223224d054246d796a20ded0415a012" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34761 } } peers { permanent_uuid: "9c5340a31b954a05aa99bb41395ed290" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34579 } }
I20260924 04:08:31.528609 2728 tablet_bootstrap.cc:492] T 3a9f9e4e4d6742318b3bf564cf2436bf P 8c9d6c009d9642109c86c0cd26db2f50: No bootstrap required, opened a new log
I20260924 04:08:31.528744 2726 ts_tablet_manager.cc:1428] T 3a9f9e4e4d6742318b3bf564cf2436bf P b223224d054246d796a20ded0415a012: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260924 04:08:31.528764 2728 ts_tablet_manager.cc:1397] T 3a9f9e4e4d6742318b3bf564cf2436bf P 8c9d6c009d9642109c86c0cd26db2f50: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 04:08:31.528894 2727 tablet_bootstrap.cc:492] T 3a9f9e4e4d6742318b3bf564cf2436bf P 9c5340a31b954a05aa99bb41395ed290: No bootstrap required, opened a new log
I20260924 04:08:31.529004 2727 ts_tablet_manager.cc:1397] T 3a9f9e4e4d6742318b3bf564cf2436bf P 9c5340a31b954a05aa99bb41395ed290: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 04:08:31.529156 2728 raft_consensus.cc:348] T 3a9f9e4e4d6742318b3bf564cf2436bf P 8c9d6c009d9642109c86c0cd26db2f50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c9d6c009d9642109c86c0cd26db2f50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46481 } } peers { permanent_uuid: "b223224d054246d796a20ded0415a012" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34761 } } peers { permanent_uuid: "9c5340a31b954a05aa99bb41395ed290" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34579 } }
I20260924 04:08:31.529279 2728 raft_consensus.cc:374] T 3a9f9e4e4d6742318b3bf564cf2436bf P 8c9d6c009d9642109c86c0cd26db2f50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:08:31.529357 2728 raft_consensus.cc:729] T 3a9f9e4e4d6742318b3bf564cf2436bf P 8c9d6c009d9642109c86c0cd26db2f50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c9d6c009d9642109c86c0cd26db2f50, State: Initialized, Role: FOLLOWER
I20260924 04:08:31.529429 2727 raft_consensus.cc:348] T 3a9f9e4e4d6742318b3bf564cf2436bf P 9c5340a31b954a05aa99bb41395ed290 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c9d6c009d9642109c86c0cd26db2f50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46481 } } peers { permanent_uuid: "b223224d054246d796a20ded0415a012" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34761 } } peers { permanent_uuid: "9c5340a31b954a05aa99bb41395ed290" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34579 } }
I20260924 04:08:31.529556 2727 raft_consensus.cc:374] T 3a9f9e4e4d6742318b3bf564cf2436bf P 9c5340a31b954a05aa99bb41395ed290 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:08:31.529510 2728 consensus_queue.cc:260] T 3a9f9e4e4d6742318b3bf564cf2436bf P 8c9d6c009d9642109c86c0cd26db2f50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c9d6c009d9642109c86c0cd26db2f50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46481 } } peers { permanent_uuid: "b223224d054246d796a20ded0415a012" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34761 } } peers { permanent_uuid: "9c5340a31b954a05aa99bb41395ed290" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34579 } }
I20260924 04:08:31.529623 2727 raft_consensus.cc:729] T 3a9f9e4e4d6742318b3bf564cf2436bf P 9c5340a31b954a05aa99bb41395ed290 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c5340a31b954a05aa99bb41395ed290, State: Initialized, Role: FOLLOWER
I20260924 04:08:31.529780 2727 consensus_queue.cc:260] T 3a9f9e4e4d6742318b3bf564cf2436bf P 9c5340a31b954a05aa99bb41395ed290 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c9d6c009d9642109c86c0cd26db2f50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46481 } } peers { permanent_uuid: "b223224d054246d796a20ded0415a012" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34761 } } peers { permanent_uuid: "9c5340a31b954a05aa99bb41395ed290" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34579 } }
I20260924 04:08:31.529850 2728 ts_tablet_manager.cc:1428] T 3a9f9e4e4d6742318b3bf564cf2436bf P 8c9d6c009d9642109c86c0cd26db2f50: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:08:31.530050 2727 ts_tablet_manager.cc:1428] T 3a9f9e4e4d6742318b3bf564cf2436bf P 9c5340a31b954a05aa99bb41395ed290: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:08:31.544981 2707 raft_consensus.cc:482] T 3a9f9e4e4d6742318b3bf564cf2436bf P 8c9d6c009d9642109c86c0cd26db2f50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:08:31.545115 2707 raft_consensus.cc:504] T 3a9f9e4e4d6742318b3bf564cf2436bf P 8c9d6c009d9642109c86c0cd26db2f50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c9d6c009d9642109c86c0cd26db2f50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46481 } } peers { permanent_uuid: "b223224d054246d796a20ded0415a012" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34761 } } peers { permanent_uuid: "9c5340a31b954a05aa99bb41395ed290" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34579 } }
I20260924 04:08:31.545632 2707 leader_election.cc:288] T 3a9f9e4e4d6742318b3bf564cf2436bf P 8c9d6c009d9642109c86c0cd26db2f50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b223224d054246d796a20ded0415a012 (127.0.98.2:34761), 9c5340a31b954a05aa99bb41395ed290 (127.0.98.3:34579)
I20260924 04:08:31.545985 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a9f9e4e4d6742318b3bf564cf2436bf" candidate_uuid: "8c9d6c009d9642109c86c0cd26db2f50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b223224d054246d796a20ded0415a012" is_pre_election: true
I20260924 04:08:31.546023 2566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a9f9e4e4d6742318b3bf564cf2436bf" candidate_uuid: "8c9d6c009d9642109c86c0cd26db2f50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c5340a31b954a05aa99bb41395ed290" is_pre_election: true
I20260924 04:08:31.546165 2419 raft_consensus.cc:2454] T 3a9f9e4e4d6742318b3bf564cf2436bf P b223224d054246d796a20ded0415a012 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c9d6c009d9642109c86c0cd26db2f50 in term 0.
I20260924 04:08:31.546181 2566 raft_consensus.cc:2454] T 3a9f9e4e4d6742318b3bf564cf2436bf P 9c5340a31b954a05aa99bb41395ed290 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c9d6c009d9642109c86c0cd26db2f50 in term 0.
I20260924 04:08:31.546460 2217 leader_election.cc:302] T 3a9f9e4e4d6742318b3bf564cf2436bf P 8c9d6c009d9642109c86c0cd26db2f50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c9d6c009d9642109c86c0cd26db2f50, b223224d054246d796a20ded0415a012; no voters:
I20260924 04:08:31.546641 2707 raft_consensus.cc:2781] T 3a9f9e4e4d6742318b3bf564cf2436bf P 8c9d6c009d9642109c86c0cd26db2f50 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:08:31.546702 2707 raft_consensus.cc:482] T 3a9f9e4e4d6742318b3bf564cf2436bf P 8c9d6c009d9642109c86c0cd26db2f50 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:08:31.546751 2707 raft_consensus.cc:3037] T 3a9f9e4e4d6742318b3bf564cf2436bf P 8c9d6c009d9642109c86c0cd26db2f50 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:08:31.547694 2707 raft_consensus.cc:504] T 3a9f9e4e4d6742318b3bf564cf2436bf P 8c9d6c009d9642109c86c0cd26db2f50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c9d6c009d9642109c86c0cd26db2f50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46481 } } peers { permanent_uuid: "b223224d054246d796a20ded0415a012" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34761 } } peers { permanent_uuid: "9c5340a31b954a05aa99bb41395ed290" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34579 } }
I20260924 04:08:31.548048 2707 leader_election.cc:288] T 3a9f9e4e4d6742318b3bf564cf2436bf P 8c9d6c009d9642109c86c0cd26db2f50 [CANDIDATE]: Term 1 election: Requested vote from peers b223224d054246d796a20ded0415a012 (127.0.98.2:34761), 9c5340a31b954a05aa99bb41395ed290 (127.0.98.3:34579)
I20260924 04:08:31.548372 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a9f9e4e4d6742318b3bf564cf2436bf" candidate_uuid: "8c9d6c009d9642109c86c0cd26db2f50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b223224d054246d796a20ded0415a012"
I20260924 04:08:31.548375 2566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a9f9e4e4d6742318b3bf564cf2436bf" candidate_uuid: "8c9d6c009d9642109c86c0cd26db2f50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c5340a31b954a05aa99bb41395ed290"
I20260924 04:08:31.548470 2419 raft_consensus.cc:3037] T 3a9f9e4e4d6742318b3bf564cf2436bf P b223224d054246d796a20ded0415a012 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:08:31.548520 2566 raft_consensus.cc:3037] T 3a9f9e4e4d6742318b3bf564cf2436bf P 9c5340a31b954a05aa99bb41395ed290 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:08:31.549358 2419 raft_consensus.cc:2454] T 3a9f9e4e4d6742318b3bf564cf2436bf P b223224d054246d796a20ded0415a012 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c9d6c009d9642109c86c0cd26db2f50 in term 1.
I20260924 04:08:31.549489 2566 raft_consensus.cc:2454] T 3a9f9e4e4d6742318b3bf564cf2436bf P 9c5340a31b954a05aa99bb41395ed290 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c9d6c009d9642109c86c0cd26db2f50 in term 1.
I20260924 04:08:31.549641 2217 leader_election.cc:302] T 3a9f9e4e4d6742318b3bf564cf2436bf P 8c9d6c009d9642109c86c0cd26db2f50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c9d6c009d9642109c86c0cd26db2f50, b223224d054246d796a20ded0415a012; no voters:
I20260924 04:08:31.549829 2707 raft_consensus.cc:2781] T 3a9f9e4e4d6742318b3bf564cf2436bf P 8c9d6c009d9642109c86c0cd26db2f50 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:08:31.549937 2707 raft_consensus.cc:686] T 3a9f9e4e4d6742318b3bf564cf2436bf P 8c9d6c009d9642109c86c0cd26db2f50 [term 1 LEADER]: Becoming Leader. State: Replica: 8c9d6c009d9642109c86c0cd26db2f50, State: Running, Role: LEADER
I20260924 04:08:31.550098 2707 consensus_queue.cc:237] T 3a9f9e4e4d6742318b3bf564cf2436bf P 8c9d6c009d9642109c86c0cd26db2f50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c9d6c009d9642109c86c0cd26db2f50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46481 } } peers { permanent_uuid: "b223224d054246d796a20ded0415a012" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34761 } } peers { permanent_uuid: "9c5340a31b954a05aa99bb41395ed290" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34579 } }
I20260924 04:08:31.551615 2129 catalog_manager.cc:5495] T 3a9f9e4e4d6742318b3bf564cf2436bf P 8c9d6c009d9642109c86c0cd26db2f50 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8c9d6c009d9642109c86c0cd26db2f50 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "8c9d6c009d9642109c86c0cd26db2f50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c9d6c009d9642109c86c0cd26db2f50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b223224d054246d796a20ded0415a012" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c5340a31b954a05aa99bb41395ed290" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34579 } health_report { overall_health: UNKNOWN } } }
I20260924 04:08:31.572887 2735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:31.569619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58194 (local address 127.0.98.1:46481)
0924 04:08:31.569883 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:31.569899 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:31.569910 (+ 11us) server_negotiation.cc:407] Connection header received
0924 04:08:31.569944 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:31.569946 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:31.569984 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:31.570073 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:08:31.570324 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:31.570935 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:31.571814 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:31.572053 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:31.572158 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:31.572422 (+ 264us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:08:31.572500 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:08:31.572580 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:31.572669 (+ 89us) server_negotiation.cc:299] Negotiation successful
0924 04:08:31.572740 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":93,"threads_started":1}
I20260924 04:08:31.578526 2566 raft_consensus.cc:1261] T 3a9f9e4e4d6742318b3bf564cf2436bf P 9c5340a31b954a05aa99bb41395ed290 [term 1 FOLLOWER]: Refusing update from remote peer 8c9d6c009d9642109c86c0cd26db2f50: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:08:31.578588 2419 raft_consensus.cc:1261] T 3a9f9e4e4d6742318b3bf564cf2436bf P b223224d054246d796a20ded0415a012 [term 1 FOLLOWER]: Refusing update from remote peer 8c9d6c009d9642109c86c0cd26db2f50: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:08:31.578908 2693 consensus_queue.cc:1035] T 3a9f9e4e4d6742318b3bf564cf2436bf P 8c9d6c009d9642109c86c0cd26db2f50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c5340a31b954a05aa99bb41395ed290" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:08:31.579062 2707 consensus_queue.cc:1035] T 3a9f9e4e4d6742318b3bf564cf2436bf P 8c9d6c009d9642109c86c0cd26db2f50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b223224d054246d796a20ded0415a012" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:08:31.607553 2129 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55234:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:08:31.607703 2129 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55234:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:08:31.609653 2129 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e422a7c9985241bf963fe50b6dedba65: Sending DeleteTablet for 3 replicas of tablet 3a9f9e4e4d6742318b3bf564cf2436bf
I20260924 04:08:31.610165 2266 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a9f9e4e4d6742318b3bf564cf2436bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:08:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53524
I20260924 04:08:31.610302 2406 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a9f9e4e4d6742318b3bf564cf2436bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:08:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56744
I20260924 04:08:31.610360 2545 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a9f9e4e4d6742318b3bf564cf2436bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:08:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47332
I20260924 04:08:31.610499 2715 tablet_replica.cc:330] T 3a9f9e4e4d6742318b3bf564cf2436bf P 8c9d6c009d9642109c86c0cd26db2f50: stopping tablet replica
I20260924 04:08:31.610530 2717 tablet_replica.cc:330] T 3a9f9e4e4d6742318b3bf564cf2436bf P 9c5340a31b954a05aa99bb41395ed290: stopping tablet replica
I20260924 04:08:31.610631 2715 raft_consensus.cc:2229] T 3a9f9e4e4d6742318b3bf564cf2436bf P 8c9d6c009d9642109c86c0cd26db2f50 [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:08:31.610662 2717 raft_consensus.cc:2229] T 3a9f9e4e4d6742318b3bf564cf2436bf P 9c5340a31b954a05aa99bb41395ed290 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:08:31.610774 2717 raft_consensus.cc:2258] T 3a9f9e4e4d6742318b3bf564cf2436bf P 9c5340a31b954a05aa99bb41395ed290 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:08:31.610821 2715 raft_consensus.cc:2258] T 3a9f9e4e4d6742318b3bf564cf2436bf P 8c9d6c009d9642109c86c0cd26db2f50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:08:31.611027 2716 tablet_replica.cc:330] T 3a9f9e4e4d6742318b3bf564cf2436bf P b223224d054246d796a20ded0415a012: stopping tablet replica
I20260924 04:08:31.611018 2129 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55234:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 04:08:31.611166 2716 raft_consensus.cc:2229] T 3a9f9e4e4d6742318b3bf564cf2436bf P b223224d054246d796a20ded0415a012 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:08:31.611282 2716 raft_consensus.cc:2258] T 3a9f9e4e4d6742318b3bf564cf2436bf P b223224d054246d796a20ded0415a012 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260924 04:08:31.611351 2129 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:08:31.611480 2717 ts_tablet_manager.cc:1905] T 3a9f9e4e4d6742318b3bf564cf2436bf P 9c5340a31b954a05aa99bb41395ed290: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:08:31.612118 2716 ts_tablet_manager.cc:1905] T 3a9f9e4e4d6742318b3bf564cf2436bf P b223224d054246d796a20ded0415a012: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:08:31.612128 2715 ts_tablet_manager.cc:1905] T 3a9f9e4e4d6742318b3bf564cf2436bf P 8c9d6c009d9642109c86c0cd26db2f50: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:08:31.614066 2716 ts_tablet_manager.cc:1918] T 3a9f9e4e4d6742318b3bf564cf2436bf P b223224d054246d796a20ded0415a012: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 04:08:31.614168 2716 log.cc:1196] T 3a9f9e4e4d6742318b3bf564cf2436bf P b223224d054246d796a20ded0415a012: Deleting WAL directory at /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/minicluster-data/ts-1/wal/wals/3a9f9e4e4d6742318b3bf564cf2436bf
I20260924 04:08:31.614176 2717 ts_tablet_manager.cc:1918] T 3a9f9e4e4d6742318b3bf564cf2436bf P 9c5340a31b954a05aa99bb41395ed290: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 04:08:31.614241 2717 log.cc:1196] T 3a9f9e4e4d6742318b3bf564cf2436bf P 9c5340a31b954a05aa99bb41395ed290: Deleting WAL directory at /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/minicluster-data/ts-2/wal/wals/3a9f9e4e4d6742318b3bf564cf2436bf
I20260924 04:08:31.614419 2716 ts_tablet_manager.cc:1939] T 3a9f9e4e4d6742318b3bf564cf2436bf P b223224d054246d796a20ded0415a012: Deleting consensus metadata
I20260924 04:08:31.614488 2717 ts_tablet_manager.cc:1939] T 3a9f9e4e4d6742318b3bf564cf2436bf P 9c5340a31b954a05aa99bb41395ed290: Deleting consensus metadata
I20260924 04:08:31.615016 2113 catalog_manager.cc:4841] TS b223224d054246d796a20ded0415a012 (127.0.98.2:34761): tablet 3a9f9e4e4d6742318b3bf564cf2436bf (table test-table [id=37d55f5cce3e47d6934c262058aed7f7]) successfully deleted
I20260924 04:08:31.615163 2115 catalog_manager.cc:4841] TS 9c5340a31b954a05aa99bb41395ed290 (127.0.98.3:34579): tablet 3a9f9e4e4d6742318b3bf564cf2436bf (table test-table [id=37d55f5cce3e47d6934c262058aed7f7]) successfully deleted
I20260924 04:08:31.615294 2715 ts_tablet_manager.cc:1918] T 3a9f9e4e4d6742318b3bf564cf2436bf P 8c9d6c009d9642109c86c0cd26db2f50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 04:08:31.615384 2715 log.cc:1196] T 3a9f9e4e4d6742318b3bf564cf2436bf P 8c9d6c009d9642109c86c0cd26db2f50: Deleting WAL directory at /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/minicluster-data/ts-0/wal/wals/3a9f9e4e4d6742318b3bf564cf2436bf
I20260924 04:08:31.615640 2715 ts_tablet_manager.cc:1939] T 3a9f9e4e4d6742318b3bf564cf2436bf P 8c9d6c009d9642109c86c0cd26db2f50: Deleting consensus metadata
I20260924 04:08:31.616405 2113 catalog_manager.cc:4841] TS 8c9d6c009d9642109c86c0cd26db2f50 (127.0.98.1:46481): tablet 3a9f9e4e4d6742318b3bf564cf2436bf (table test-table [id=37d55f5cce3e47d6934c262058aed7f7]) successfully deleted
I20260924 04:08:31.618955 2266 tablet_service.cc:1467] Processing CreateTablet for tablet 83e7aa2ea5ac408a876bf42d259c80e1 (DEFAULT_TABLE table=test-table [id=5dcdd54f9f3e47448c8743335373edc2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:08:31.618988 2406 tablet_service.cc:1467] Processing CreateTablet for tablet 83e7aa2ea5ac408a876bf42d259c80e1 (DEFAULT_TABLE table=test-table [id=5dcdd54f9f3e47448c8743335373edc2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:08:31.619010 2545 tablet_service.cc:1467] Processing CreateTablet for tablet 83e7aa2ea5ac408a876bf42d259c80e1 (DEFAULT_TABLE table=test-table [id=5dcdd54f9f3e47448c8743335373edc2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:08:31.619225 2406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83e7aa2ea5ac408a876bf42d259c80e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:08:31.619228 2545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83e7aa2ea5ac408a876bf42d259c80e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:08:31.619227 2266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83e7aa2ea5ac408a876bf42d259c80e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:08:31.621014 2728 tablet_bootstrap.cc:492] T 83e7aa2ea5ac408a876bf42d259c80e1 P 8c9d6c009d9642109c86c0cd26db2f50: Bootstrap starting.
I20260924 04:08:31.621356 2727 tablet_bootstrap.cc:492] T 83e7aa2ea5ac408a876bf42d259c80e1 P 9c5340a31b954a05aa99bb41395ed290: Bootstrap starting.
I20260924 04:08:31.621428 2726 tablet_bootstrap.cc:492] T 83e7aa2ea5ac408a876bf42d259c80e1 P b223224d054246d796a20ded0415a012: Bootstrap starting.
I20260924 04:08:31.621824 2728 tablet_bootstrap.cc:654] T 83e7aa2ea5ac408a876bf42d259c80e1 P 8c9d6c009d9642109c86c0cd26db2f50: Neither blocks nor log segments found. Creating new log.
I20260924 04:08:31.622197 2727 tablet_bootstrap.cc:654] T 83e7aa2ea5ac408a876bf42d259c80e1 P 9c5340a31b954a05aa99bb41395ed290: Neither blocks nor log segments found. Creating new log.
I20260924 04:08:31.622771 2726 tablet_bootstrap.cc:654] T 83e7aa2ea5ac408a876bf42d259c80e1 P b223224d054246d796a20ded0415a012: Neither blocks nor log segments found. Creating new log.
I20260924 04:08:31.623188 2728 tablet_bootstrap.cc:492] T 83e7aa2ea5ac408a876bf42d259c80e1 P 8c9d6c009d9642109c86c0cd26db2f50: No bootstrap required, opened a new log
I20260924 04:08:31.623273 2728 ts_tablet_manager.cc:1397] T 83e7aa2ea5ac408a876bf42d259c80e1 P 8c9d6c009d9642109c86c0cd26db2f50: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 04:08:31.623376 2727 tablet_bootstrap.cc:492] T 83e7aa2ea5ac408a876bf42d259c80e1 P 9c5340a31b954a05aa99bb41395ed290: No bootstrap required, opened a new log
I20260924 04:08:31.623442 2727 ts_tablet_manager.cc:1397] T 83e7aa2ea5ac408a876bf42d259c80e1 P 9c5340a31b954a05aa99bb41395ed290: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 04:08:31.623638 2728 raft_consensus.cc:348] T 83e7aa2ea5ac408a876bf42d259c80e1 P 8c9d6c009d9642109c86c0cd26db2f50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c9d6c009d9642109c86c0cd26db2f50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46481 } } peers { permanent_uuid: "b223224d054246d796a20ded0415a012" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34761 } } peers { permanent_uuid: "9c5340a31b954a05aa99bb41395ed290" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34579 } }
I20260924 04:08:31.623752 2728 raft_consensus.cc:374] T 83e7aa2ea5ac408a876bf42d259c80e1 P 8c9d6c009d9642109c86c0cd26db2f50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:08:31.623798 2728 raft_consensus.cc:729] T 83e7aa2ea5ac408a876bf42d259c80e1 P 8c9d6c009d9642109c86c0cd26db2f50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c9d6c009d9642109c86c0cd26db2f50, State: Initialized, Role: FOLLOWER
I20260924 04:08:31.623795 2727 raft_consensus.cc:348] T 83e7aa2ea5ac408a876bf42d259c80e1 P 9c5340a31b954a05aa99bb41395ed290 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c9d6c009d9642109c86c0cd26db2f50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46481 } } peers { permanent_uuid: "b223224d054246d796a20ded0415a012" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34761 } } peers { permanent_uuid: "9c5340a31b954a05aa99bb41395ed290" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34579 } }
I20260924 04:08:31.623893 2727 raft_consensus.cc:374] T 83e7aa2ea5ac408a876bf42d259c80e1 P 9c5340a31b954a05aa99bb41395ed290 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:08:31.623924 2727 raft_consensus.cc:729] T 83e7aa2ea5ac408a876bf42d259c80e1 P 9c5340a31b954a05aa99bb41395ed290 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c5340a31b954a05aa99bb41395ed290, State: Initialized, Role: FOLLOWER
I20260924 04:08:31.623924 2728 consensus_queue.cc:260] T 83e7aa2ea5ac408a876bf42d259c80e1 P 8c9d6c009d9642109c86c0cd26db2f50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c9d6c009d9642109c86c0cd26db2f50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46481 } } peers { permanent_uuid: "b223224d054246d796a20ded0415a012" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34761 } } peers { permanent_uuid: "9c5340a31b954a05aa99bb41395ed290" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34579 } }
I20260924 04:08:31.624033 2727 consensus_queue.cc:260] T 83e7aa2ea5ac408a876bf42d259c80e1 P 9c5340a31b954a05aa99bb41395ed290 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c9d6c009d9642109c86c0cd26db2f50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46481 } } peers { permanent_uuid: "b223224d054246d796a20ded0415a012" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34761 } } peers { permanent_uuid: "9c5340a31b954a05aa99bb41395ed290" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34579 } }
I20260924 04:08:31.624085 2726 tablet_bootstrap.cc:492] T 83e7aa2ea5ac408a876bf42d259c80e1 P b223224d054246d796a20ded0415a012: No bootstrap required, opened a new log
I20260924 04:08:31.624154 2726 ts_tablet_manager.cc:1397] T 83e7aa2ea5ac408a876bf42d259c80e1 P b223224d054246d796a20ded0415a012: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 04:08:31.624177 2728 ts_tablet_manager.cc:1428] T 83e7aa2ea5ac408a876bf42d259c80e1 P 8c9d6c009d9642109c86c0cd26db2f50: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:08:31.624264 2727 ts_tablet_manager.cc:1428] T 83e7aa2ea5ac408a876bf42d259c80e1 P 9c5340a31b954a05aa99bb41395ed290: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:08:31.624524 2726 raft_consensus.cc:348] T 83e7aa2ea5ac408a876bf42d259c80e1 P b223224d054246d796a20ded0415a012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c9d6c009d9642109c86c0cd26db2f50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46481 } } peers { permanent_uuid: "b223224d054246d796a20ded0415a012" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34761 } } peers { permanent_uuid: "9c5340a31b954a05aa99bb41395ed290" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34579 } }
I20260924 04:08:31.624631 2726 raft_consensus.cc:374] T 83e7aa2ea5ac408a876bf42d259c80e1 P b223224d054246d796a20ded0415a012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:08:31.624665 2726 raft_consensus.cc:729] T 83e7aa2ea5ac408a876bf42d259c80e1 P b223224d054246d796a20ded0415a012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b223224d054246d796a20ded0415a012, State: Initialized, Role: FOLLOWER
I20260924 04:08:31.624809 2726 consensus_queue.cc:260] T 83e7aa2ea5ac408a876bf42d259c80e1 P b223224d054246d796a20ded0415a012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c9d6c009d9642109c86c0cd26db2f50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46481 } } peers { permanent_uuid: "b223224d054246d796a20ded0415a012" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34761 } } peers { permanent_uuid: "9c5340a31b954a05aa99bb41395ed290" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34579 } }
I20260924 04:08:31.625172 2726 ts_tablet_manager.cc:1428] T 83e7aa2ea5ac408a876bf42d259c80e1 P b223224d054246d796a20ded0415a012: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260924 04:08:31.833951 2702 raft_consensus.cc:482] T 83e7aa2ea5ac408a876bf42d259c80e1 P 9c5340a31b954a05aa99bb41395ed290 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:08:31.834161 2702 raft_consensus.cc:504] T 83e7aa2ea5ac408a876bf42d259c80e1 P 9c5340a31b954a05aa99bb41395ed290 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c9d6c009d9642109c86c0cd26db2f50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46481 } } peers { permanent_uuid: "b223224d054246d796a20ded0415a012" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34761 } } peers { permanent_uuid: "9c5340a31b954a05aa99bb41395ed290" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34579 } }
I20260924 04:08:31.834597 2702 leader_election.cc:288] T 83e7aa2ea5ac408a876bf42d259c80e1 P 9c5340a31b954a05aa99bb41395ed290 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c9d6c009d9642109c86c0cd26db2f50 (127.0.98.1:46481), b223224d054246d796a20ded0415a012 (127.0.98.2:34761)
I20260924 04:08:31.835120 2283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83e7aa2ea5ac408a876bf42d259c80e1" candidate_uuid: "9c5340a31b954a05aa99bb41395ed290" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c9d6c009d9642109c86c0cd26db2f50" is_pre_election: true
I20260924 04:08:31.835192 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83e7aa2ea5ac408a876bf42d259c80e1" candidate_uuid: "9c5340a31b954a05aa99bb41395ed290" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b223224d054246d796a20ded0415a012" is_pre_election: true
I20260924 04:08:31.835318 2283 raft_consensus.cc:2454] T 83e7aa2ea5ac408a876bf42d259c80e1 P 8c9d6c009d9642109c86c0cd26db2f50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c5340a31b954a05aa99bb41395ed290 in term 0.
I20260924 04:08:31.835403 2419 raft_consensus.cc:2454] T 83e7aa2ea5ac408a876bf42d259c80e1 P b223224d054246d796a20ded0415a012 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c5340a31b954a05aa99bb41395ed290 in term 0.
I20260924 04:08:31.835734 2497 leader_election.cc:302] T 83e7aa2ea5ac408a876bf42d259c80e1 P 9c5340a31b954a05aa99bb41395ed290 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c9d6c009d9642109c86c0cd26db2f50, 9c5340a31b954a05aa99bb41395ed290; no voters:
I20260924 04:08:31.835913 2702 raft_consensus.cc:2781] T 83e7aa2ea5ac408a876bf42d259c80e1 P 9c5340a31b954a05aa99bb41395ed290 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:08:31.835971 2702 raft_consensus.cc:482] T 83e7aa2ea5ac408a876bf42d259c80e1 P 9c5340a31b954a05aa99bb41395ed290 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:08:31.836010 2702 raft_consensus.cc:3037] T 83e7aa2ea5ac408a876bf42d259c80e1 P 9c5340a31b954a05aa99bb41395ed290 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:08:31.837128 2702 raft_consensus.cc:504] T 83e7aa2ea5ac408a876bf42d259c80e1 P 9c5340a31b954a05aa99bb41395ed290 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c9d6c009d9642109c86c0cd26db2f50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46481 } } peers { permanent_uuid: "b223224d054246d796a20ded0415a012" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34761 } } peers { permanent_uuid: "9c5340a31b954a05aa99bb41395ed290" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34579 } }
I20260924 04:08:31.837442 2702 leader_election.cc:288] T 83e7aa2ea5ac408a876bf42d259c80e1 P 9c5340a31b954a05aa99bb41395ed290 [CANDIDATE]: Term 1 election: Requested vote from peers 8c9d6c009d9642109c86c0cd26db2f50 (127.0.98.1:46481), b223224d054246d796a20ded0415a012 (127.0.98.2:34761)
I20260924 04:08:31.837808 2283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83e7aa2ea5ac408a876bf42d259c80e1" candidate_uuid: "9c5340a31b954a05aa99bb41395ed290" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c9d6c009d9642109c86c0cd26db2f50"
I20260924 04:08:31.837854 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83e7aa2ea5ac408a876bf42d259c80e1" candidate_uuid: "9c5340a31b954a05aa99bb41395ed290" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b223224d054246d796a20ded0415a012"
I20260924 04:08:31.837935 2283 raft_consensus.cc:3037] T 83e7aa2ea5ac408a876bf42d259c80e1 P 8c9d6c009d9642109c86c0cd26db2f50 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:08:31.837954 2419 raft_consensus.cc:3037] T 83e7aa2ea5ac408a876bf42d259c80e1 P b223224d054246d796a20ded0415a012 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:08:31.839040 2283 raft_consensus.cc:2454] T 83e7aa2ea5ac408a876bf42d259c80e1 P 8c9d6c009d9642109c86c0cd26db2f50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c5340a31b954a05aa99bb41395ed290 in term 1.
I20260924 04:08:31.839040 2419 raft_consensus.cc:2454] T 83e7aa2ea5ac408a876bf42d259c80e1 P b223224d054246d796a20ded0415a012 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c5340a31b954a05aa99bb41395ed290 in term 1.
I20260924 04:08:31.839327 2497 leader_election.cc:302] T 83e7aa2ea5ac408a876bf42d259c80e1 P 9c5340a31b954a05aa99bb41395ed290 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c9d6c009d9642109c86c0cd26db2f50, 9c5340a31b954a05aa99bb41395ed290; no voters:
I20260924 04:08:31.839479 2702 raft_consensus.cc:2781] T 83e7aa2ea5ac408a876bf42d259c80e1 P 9c5340a31b954a05aa99bb41395ed290 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:08:31.839564 2702 raft_consensus.cc:686] T 83e7aa2ea5ac408a876bf42d259c80e1 P 9c5340a31b954a05aa99bb41395ed290 [term 1 LEADER]: Becoming Leader. State: Replica: 9c5340a31b954a05aa99bb41395ed290, State: Running, Role: LEADER
I20260924 04:08:31.839725 2702 consensus_queue.cc:237] T 83e7aa2ea5ac408a876bf42d259c80e1 P 9c5340a31b954a05aa99bb41395ed290 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c9d6c009d9642109c86c0cd26db2f50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46481 } } peers { permanent_uuid: "b223224d054246d796a20ded0415a012" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34761 } } peers { permanent_uuid: "9c5340a31b954a05aa99bb41395ed290" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34579 } }
I20260924 04:08:31.841530 2129 catalog_manager.cc:5495] T 83e7aa2ea5ac408a876bf42d259c80e1 P 9c5340a31b954a05aa99bb41395ed290 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c5340a31b954a05aa99bb41395ed290 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "9c5340a31b954a05aa99bb41395ed290" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c9d6c009d9642109c86c0cd26db2f50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b223224d054246d796a20ded0415a012" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c5340a31b954a05aa99bb41395ed290" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34579 } health_report { overall_health: HEALTHY } } }
I20260924 04:08:31.866520 2744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:31.863466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60732 (local address 127.0.98.3:34579)
0924 04:08:31.863759 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:31.863766 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:31.863778 (+ 12us) server_negotiation.cc:407] Connection header received
0924 04:08:31.863814 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:31.863816 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:31.863855 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:31.863936 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:08:31.864144 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:31.864661 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:31.865579 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:31.865813 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:31.865900 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:31.866151 (+ 251us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:08:31.866226 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:08:31.866279 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:31.866359 (+ 80us) server_negotiation.cc:299] Negotiation successful
0924 04:08:31.866403 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":130,"threads_started":1}
I20260924 04:08:31.873068 2283 raft_consensus.cc:1261] T 83e7aa2ea5ac408a876bf42d259c80e1 P 8c9d6c009d9642109c86c0cd26db2f50 [term 1 FOLLOWER]: Refusing update from remote peer 9c5340a31b954a05aa99bb41395ed290: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:08:31.873149 2419 raft_consensus.cc:1261] T 83e7aa2ea5ac408a876bf42d259c80e1 P b223224d054246d796a20ded0415a012 [term 1 FOLLOWER]: Refusing update from remote peer 9c5340a31b954a05aa99bb41395ed290: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:08:31.873507 2697 consensus_queue.cc:1035] T 83e7aa2ea5ac408a876bf42d259c80e1 P 9c5340a31b954a05aa99bb41395ed290 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b223224d054246d796a20ded0415a012" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:08:31.873700 2702 consensus_queue.cc:1035] T 83e7aa2ea5ac408a876bf42d259c80e1 P 9c5340a31b954a05aa99bb41395ed290 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c9d6c009d9642109c86c0cd26db2f50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 04:08:31.885774 2695 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 04:08:33.436626 2126 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55234:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:08:33.436858 2126 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55234:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:08:33.439481 2126 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e422a7c9985241bf963fe50b6dedba65: Sending DeleteTablet for 3 replicas of tablet 83e7aa2ea5ac408a876bf42d259c80e1
I20260924 04:08:33.440110 2266 tablet_service.cc:1514] Processing DeleteTablet for tablet 83e7aa2ea5ac408a876bf42d259c80e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:08:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53524
I20260924 04:08:33.440147 2406 tablet_service.cc:1514] Processing DeleteTablet for tablet 83e7aa2ea5ac408a876bf42d259c80e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:08:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56744
I20260924 04:08:33.440183 2545 tablet_service.cc:1514] Processing DeleteTablet for tablet 83e7aa2ea5ac408a876bf42d259c80e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:08:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47332
I20260924 04:08:33.440508 2771 tablet_replica.cc:330] T 83e7aa2ea5ac408a876bf42d259c80e1 P 9c5340a31b954a05aa99bb41395ed290: stopping tablet replica
I20260924 04:08:33.440506 2770 tablet_replica.cc:330] T 83e7aa2ea5ac408a876bf42d259c80e1 P b223224d054246d796a20ded0415a012: stopping tablet replica
I20260924 04:08:33.440575 2769 tablet_replica.cc:330] T 83e7aa2ea5ac408a876bf42d259c80e1 P 8c9d6c009d9642109c86c0cd26db2f50: stopping tablet replica
I20260924 04:08:33.440650 2770 raft_consensus.cc:2229] T 83e7aa2ea5ac408a876bf42d259c80e1 P b223224d054246d796a20ded0415a012 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:08:33.440725 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 2201
I20260924 04:08:33.440716 2771 raft_consensus.cc:2229] T 83e7aa2ea5ac408a876bf42d259c80e1 P 9c5340a31b954a05aa99bb41395ed290 [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:08:33.441424 2771 raft_consensus.cc:2258] T 83e7aa2ea5ac408a876bf42d259c80e1 P 9c5340a31b954a05aa99bb41395ed290 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:08:33.442312 2771 ts_tablet_manager.cc:1905] T 83e7aa2ea5ac408a876bf42d259c80e1 P 9c5340a31b954a05aa99bb41395ed290: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:08:33.442536 2770 raft_consensus.cc:2258] T 83e7aa2ea5ac408a876bf42d259c80e1 P b223224d054246d796a20ded0415a012 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:08:33.443264 2770 ts_tablet_manager.cc:1905] T 83e7aa2ea5ac408a876bf42d259c80e1 P b223224d054246d796a20ded0415a012: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:08:33.445220 2771 ts_tablet_manager.cc:1918] T 83e7aa2ea5ac408a876bf42d259c80e1 P 9c5340a31b954a05aa99bb41395ed290: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 04:08:33.445330 2771 log.cc:1196] T 83e7aa2ea5ac408a876bf42d259c80e1 P 9c5340a31b954a05aa99bb41395ed290: Deleting WAL directory at /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/minicluster-data/ts-2/wal/wals/83e7aa2ea5ac408a876bf42d259c80e1
I20260924 04:08:33.445632 2771 ts_tablet_manager.cc:1939] T 83e7aa2ea5ac408a876bf42d259c80e1 P 9c5340a31b954a05aa99bb41395ed290: Deleting consensus metadata
I20260924 04:08:33.446070 2770 ts_tablet_manager.cc:1918] T 83e7aa2ea5ac408a876bf42d259c80e1 P b223224d054246d796a20ded0415a012: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 04:08:33.446146 2770 log.cc:1196] T 83e7aa2ea5ac408a876bf42d259c80e1 P b223224d054246d796a20ded0415a012: Deleting WAL directory at /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790222898807720-392-0/minicluster-data/ts-1/wal/wals/83e7aa2ea5ac408a876bf42d259c80e1
I20260924 04:08:33.446383 2115 catalog_manager.cc:4841] TS 9c5340a31b954a05aa99bb41395ed290 (127.0.98.3:34579): tablet 83e7aa2ea5ac408a876bf42d259c80e1 (table test-table [id=5dcdd54f9f3e47448c8743335373edc2]) successfully deleted
I20260924 04:08:33.446445 2770 ts_tablet_manager.cc:1939] T 83e7aa2ea5ac408a876bf42d259c80e1 P b223224d054246d796a20ded0415a012: Deleting consensus metadata
I20260924 04:08:33.447089 2113 catalog_manager.cc:4841] TS b223224d054246d796a20ded0415a012 (127.0.98.2:34761): tablet 83e7aa2ea5ac408a876bf42d259c80e1 (table test-table [id=5dcdd54f9f3e47448c8743335373edc2]) successfully deleted
I20260924 04:08:33.449007 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 2341
W20260924 04:08:33.448998 2113 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46481 (error 108)
I20260924 04:08:33.450578 2773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:33.450192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46481 (local address 127.0.0.1:58208)
0924 04:08:33.450431 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:33.450496 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":98,"threads_started":1}
W20260924 04:08:33.450831 2113 catalog_manager.cc:4584] TS 8c9d6c009d9642109c86c0cd26db2f50 (127.0.98.1:46481): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 83e7aa2ea5ac408a876bf42d259c80e1: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46481: connect: Connection refused (error 111)
I20260924 04:08:33.456127 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 2481
I20260924 04:08:33.462798 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 2097
2026-09-24T04:08:33Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6748 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222898807720-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:08:33 dist-test-slave-00ld krb5kdc[2776](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:08:33 dist-test-slave-00ld krb5kdc[2776](info): set up 2 sockets
Sep 24 04:08:33 dist-test-slave-00ld krb5kdc[2776](info): commencing operation
krb5kdc: starting...
W20260924 04:08:34.500716 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222898807720-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222898807720-392-0/krb5kdc/test-user.keytab.
Sep 24 04:08:34 dist-test-slave-00ld krb5kdc[2776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:08:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:08:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:08:34.621609 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222898807720-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222898807720-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222898807720-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37071
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37071
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:08:34.732182 2789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:08:34.732496 2789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:08:34.732575 2789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:08:34.736585 2789 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:08:34.736687 2789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:08:34.736752 2789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:08:34.736790 2789 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:08:34.736820 2789 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:08:34.742115 2789 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222898807720-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222898807720-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37071
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2789
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222898807720-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:08:34.743494 2789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:08:34.744913 2789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:08:34.753990 2797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:08:34.753980 2794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:08:34.753971 2789 server_base.cc:1034] running on GCE node
W20260924 04:08:34.753952 2795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:08:34.754756 2789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:08:34.755867 2789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:08:34.757037 2789 hybrid_clock.cc:648] HybridClock initialized: now 1790222914757010 us; error 45 us; skew 500 ppm
Sep 24 04:08:34 dist-test-slave-00ld krb5kdc[2776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222914, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:08:34.759739 2789 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:08:34.760604 2789 webserver.cc:460] Webserver started at http://127.0.98.62:36211/ using document root <none> and password file <none>
I20260924 04:08:34.761179 2789 fs_manager.cc:359] Metadata directory not provided
I20260924 04:08:34.761272 2789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:08:34.761502 2789 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:08:34.763511 2789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222898807720-392-0/minicluster-data/master-0/data/instance:
uuid: "44c067dcc5dc40f5a0074875fed01fb6"
format_stamp: "Formatted at 2026-09-24 04:08:34 on dist-test-slave-00ld"
server_key: "58825cb4b8ea53f3463bd933754b5dc7"
server_key_iv: "3e77c15d7151e94866b1aebceae55374"
server_key_version: "encryptionkey@0"
I20260924 04:08:34.764094 2789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222898807720-392-0/minicluster-data/master-0/wal/instance:
uuid: "44c067dcc5dc40f5a0074875fed01fb6"
format_stamp: "Formatted at 2026-09-24 04:08:34 on dist-test-slave-00ld"
server_key: "58825cb4b8ea53f3463bd933754b5dc7"
server_key_iv: "3e77c15d7151e94866b1aebceae55374"
server_key_version: "encryptionkey@0"
I20260924 04:08:34.767771 2789 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260924 04:08:34.770143 2804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:34.771337 2789 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:08:34.771486 2789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222898807720-392-0/minicluster-data/master-0/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222898807720-392-0/minicluster-data/master-0/wal
uuid: "44c067dcc5dc40f5a0074875fed01fb6"
format_stamp: "Formatted at 2026-09-24 04:08:34 on dist-test-slave-00ld"
server_key: "58825cb4b8ea53f3463bd933754b5dc7"
server_key_iv: "3e77c15d7151e94866b1aebceae55374"
server_key_version: "encryptionkey@0"
I20260924 04:08:34.771610 2789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222898807720-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222898807720-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222898807720-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:08:34.792666 2789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:08:34.795819 2789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:08:34.796036 2789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:08:34.803377 2789 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37071
I20260924 04:08:34.803402 2856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37071 every 8 connection(s)
I20260924 04:08:34.804319 2789 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:08:34.807008 2857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:08:34.807626 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 2789
I20260924 04:08:34.807713 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222898807720-392-0/minicluster-data/master-0/wal/instance
I20260924 04:08:34.808033 392 external_mini_cluster.cc:1428] Setting key 72a8769e92c079d96c11f3195f6177ed
I20260924 04:08:34.813656 2857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44c067dcc5dc40f5a0074875fed01fb6: Bootstrap starting.
I20260924 04:08:34.816195 2857 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 44c067dcc5dc40f5a0074875fed01fb6: Neither blocks nor log segments found. Creating new log.
I20260924 04:08:34.817024 2857 log.cc:824] T 00000000000000000000000000000000 P 44c067dcc5dc40f5a0074875fed01fb6: Log is configured to *not* fsync() on all Append() calls
Sep 24 04:08:34 dist-test-slave-00ld krb5kdc[2776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:08:34.819451 2857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44c067dcc5dc40f5a0074875fed01fb6: No bootstrap required, opened a new log
I20260924 04:08:34.823928 2860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:34.809528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59364 (local address 127.0.98.62:37071)
0924 04:08:34.809947 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:34.809965 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:34.810011 (+ 46us) server_negotiation.cc:407] Connection header received
0924 04:08:34.810811 (+ 800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:34.810834 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:34.811266 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:34.811643 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:34.812345 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:34.813834 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:34.814716 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:34.815404 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:34.815566 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:34.818303 (+ 2737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:34.818309 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:34.818326 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:34.818356 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:34.820315 (+ 1959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:34.820755 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:34.820761 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:34.820767 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:34.820909 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:34.821432 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:34.821439 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:34.821441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:34.822061 (+ 620us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:34.822261 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:34.822561 (+ 300us) server_negotiation.cc:299] Negotiation successful
0924 04:08:34.822887 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":97,"threads_started":1}
I20260924 04:08:34.824059 2857 raft_consensus.cc:348] T 00000000000000000000000000000000 P 44c067dcc5dc40f5a0074875fed01fb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44c067dcc5dc40f5a0074875fed01fb6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37071 } }
I20260924 04:08:34.824331 2857 raft_consensus.cc:374] T 00000000000000000000000000000000 P 44c067dcc5dc40f5a0074875fed01fb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:08:34.824388 2857 raft_consensus.cc:729] T 00000000000000000000000000000000 P 44c067dcc5dc40f5a0074875fed01fb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44c067dcc5dc40f5a0074875fed01fb6, State: Initialized, Role: FOLLOWER
I20260924 04:08:34.824904 2857 consensus_queue.cc:260] T 00000000000000000000000000000000 P 44c067dcc5dc40f5a0074875fed01fb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44c067dcc5dc40f5a0074875fed01fb6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37071 } }
I20260924 04:08:34.825074 2857 raft_consensus.cc:388] T 00000000000000000000000000000000 P 44c067dcc5dc40f5a0074875fed01fb6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:08:34.825174 2857 raft_consensus.cc:482] T 00000000000000000000000000000000 P 44c067dcc5dc40f5a0074875fed01fb6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:08:34.825281 2857 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 44c067dcc5dc40f5a0074875fed01fb6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:08:34.826320 2857 raft_consensus.cc:504] T 00000000000000000000000000000000 P 44c067dcc5dc40f5a0074875fed01fb6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44c067dcc5dc40f5a0074875fed01fb6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37071 } }
I20260924 04:08:34.826658 2857 leader_election.cc:302] T 00000000000000000000000000000000 P 44c067dcc5dc40f5a0074875fed01fb6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 44c067dcc5dc40f5a0074875fed01fb6; no voters:
I20260924 04:08:34.826920 2857 leader_election.cc:288] T 00000000000000000000000000000000 P 44c067dcc5dc40f5a0074875fed01fb6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:08:34.827113 2862 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 44c067dcc5dc40f5a0074875fed01fb6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:08:34.827368 2862 raft_consensus.cc:686] T 00000000000000000000000000000000 P 44c067dcc5dc40f5a0074875fed01fb6 [term 1 LEADER]: Becoming Leader. State: Replica: 44c067dcc5dc40f5a0074875fed01fb6, State: Running, Role: LEADER
I20260924 04:08:34.827683 2862 consensus_queue.cc:237] T 00000000000000000000000000000000 P 44c067dcc5dc40f5a0074875fed01fb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44c067dcc5dc40f5a0074875fed01fb6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37071 } }
I20260924 04:08:34.828197 2857 sys_catalog.cc:564] T 00000000000000000000000000000000 P 44c067dcc5dc40f5a0074875fed01fb6 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:08:34.829486 2864 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44c067dcc5dc40f5a0074875fed01fb6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 44c067dcc5dc40f5a0074875fed01fb6. Latest consensus state: current_term: 1 leader_uuid: "44c067dcc5dc40f5a0074875fed01fb6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44c067dcc5dc40f5a0074875fed01fb6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37071 } } }
I20260924 04:08:34.829556 2863 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44c067dcc5dc40f5a0074875fed01fb6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "44c067dcc5dc40f5a0074875fed01fb6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44c067dcc5dc40f5a0074875fed01fb6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37071 } } }
I20260924 04:08:34.829627 2864 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44c067dcc5dc40f5a0074875fed01fb6 [sys.catalog]: This master's current role is: LEADER
I20260924 04:08:34.829649 2863 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44c067dcc5dc40f5a0074875fed01fb6 [sys.catalog]: This master's current role is: LEADER
I20260924 04:08:34.830801 2871 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:08:34.833988 2871 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:08:34.839709 2871 catalog_manager.cc:1347] Generated new cluster ID: 2ce13f231cef44ecb55e8a2d924df2a7
I20260924 04:08:34.839799 2871 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:08:34.849921 2871 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:08:34.850901 2871 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:08:34.857504 2871 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 44c067dcc5dc40f5a0074875fed01fb6: Generated new TSK 0
I20260924 04:08:34.858263 2871 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222898807720-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222898807720-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:08:34.893752 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222898807720-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222898807720-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222898807720-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222898807720-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37071
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222898807720-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:08:35.006877 2883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:08:35.007171 2883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:08:35.007216 2883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:08:35.011466 2883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:08:35.011538 2883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:08:35.011598 2883 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:08:35.016978 2883 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222898807720-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222898807720-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222898807720-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222898807720-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2883
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222898807720-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:08:35.018055 2883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:08:35.019110 2883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:08:35.028489 2891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:08:35.028609 2883 server_base.cc:1034] running on GCE node
W20260924 04:08:35.028489 2889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:08:35.028532 2888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:08:35.029241 2883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:08:35.029978 2883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:08:35.031159 2883 hybrid_clock.cc:648] HybridClock initialized: now 1790222915031115 us; error 62 us; skew 500 ppm
Sep 24 04:08:35 dist-test-slave-00ld krb5kdc[2776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222915, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:08:35.033978 2883 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:08:35.034832 2883 webserver.cc:460] Webserver started at http://127.0.98.1:45907/ using document root <none> and password file <none>
I20260924 04:08:35.035389 2883 fs_manager.cc:359] Metadata directory not provided
I20260924 04:08:35.035480 2883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:08:35.035722 2883 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:08:35.037796 2883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222898807720-392-0/minicluster-data/ts-0/data/instance:
uuid: "da3d2cb1a2f24b3ebd18a78ea80b763e"
format_stamp: "Formatted at 2026-09-24 04:08:35 on dist-test-slave-00ld"
server_key: "eecda5018357b6ae3f1367426f1e28c1"
server_key_iv: "21c3810381a5b449fefbd172dcbd7005"
server_key_version: "encryptionkey@0"
I20260924 04:08:35.038362 2883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222898807720-392-0/minicluster-data/ts-0/wal/instance:
uuid: "da3d2cb1a2f24b3ebd18a78ea80b763e"
format_stamp: "Formatted at 2026-09-24 04:08:35 on dist-test-slave-00ld"
server_key: "eecda5018357b6ae3f1367426f1e28c1"
server_key_iv: "21c3810381a5b449fefbd172dcbd7005"
server_key_version: "encryptionkey@0"
I20260924 04:08:35.041998 2883 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:08:35.044380 2898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:35.045630 2883 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:08:35.045766 2883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222898807720-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222898807720-392-0/minicluster-data/ts-0/wal
uuid: "da3d2cb1a2f24b3ebd18a78ea80b763e"
format_stamp: "Formatted at 2026-09-24 04:08:35 on dist-test-slave-00ld"
server_key: "eecda5018357b6ae3f1367426f1e28c1"
server_key_iv: "21c3810381a5b449fefbd172dcbd7005"
server_key_version: "encryptionkey@0"
I20260924 04:08:35.045879 2883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222898807720-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222898807720-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222898807720-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:08:35.056896 2883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:08:35.059963 2883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:08:35.060174 2883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:08:35.060840 2883 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:08:35.061688 2883 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:08:35.061735 2883 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:35.061805 2883 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:08:35.061836 2883 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:35.070142 2883 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35823
I20260924 04:08:35.070235 3011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35823 every 8 connection(s)
I20260924 04:08:35.071141 2883 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222898807720-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:08:35.080282 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 2883
I20260924 04:08:35.080415 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222898807720-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:08:35.080776 392 external_mini_cluster.cc:1428] Setting key c4e78f2ba97d9c8415394d68453402eb
Sep 24 04:08:35 dist-test-slave-00ld krb5kdc[2776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222915, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:08:35.085785 2860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:35.073029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35835 (local address 127.0.98.62:37071)
0924 04:08:35.073168 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:35.073172 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:35.074017 (+ 845us) server_negotiation.cc:407] Connection header received
0924 04:08:35.075467 (+ 1450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:35.075470 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:35.075529 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:35.075633 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:35.076863 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:35.077405 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:35.078583 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:35.078813 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:35.078927 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:35.081765 (+ 2838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:35.081769 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:35.081771 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:35.081804 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:35.083269 (+ 1465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:35.084015 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:35.084021 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:35.084023 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:35.084121 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:35.084447 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:35.084452 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:35.084454 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:35.084730 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:35.084827 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:35.085360 (+ 533us) server_negotiation.cc:299] Negotiation successful
0924 04:08:35.085519 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 04:08:35.086558 3014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:35.073381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37071 (local address 127.0.98.1:35835)
0924 04:08:35.073823 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:35.073866 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:35.075248 (+ 1382us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:35.075818 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:35.075826 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:35.076181 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:08:35.076664 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:35.076734 (+ 70us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:35.077538 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:35.077541 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:35.078439 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:35.078448 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:35.078965 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:35.078969 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:35.079173 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:35.079751 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:08:35.079767 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:08:35.081609 (+ 1842us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:08:35.083385 (+ 1776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:35.083394 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:35.083412 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:35.083800 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:35.084219 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:35.084222 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:35.084223 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:35.084355 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:35.084838 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:08:35.084843 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:08:35.085097 (+ 254us) client_negotiation.cc:769] Sending connection context
0924 04:08:35.085259 (+ 162us) client_negotiation.cc:240] Negotiation successful
0924 04:08:35.085572 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":111,"threads_started":1}
I20260924 04:08:35.087781 3012 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37071
I20260924 04:08:35.088027 3012 heartbeater.cc:464] Registering TS with master...
I20260924 04:08:35.088526 3012 heartbeater.cc:511] Master 127.0.98.62:37071 requested a full tablet report, sending...
I20260924 04:08:35.090080 2821 ts_manager.cc:194] Registered new tserver with Master: da3d2cb1a2f24b3ebd18a78ea80b763e (127.0.98.1:35823)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 04:08:35.091295 2821 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35835
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222898807720-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222898807720-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:08:35.108110 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222898807720-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222898807720-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222898807720-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222898807720-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37071
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222898807720-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:08:35.218080 3017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:08:35.218351 3017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:08:35.218395 3017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:08:35.222474 3017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:08:35.222535 3017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:08:35.222635 3017 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:08:35.228170 3017 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222898807720-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222898807720-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222898807720-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222898807720-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3017
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222898807720-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:08:35.229331 3017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:08:35.230389 3017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:08:35.240013 3017 server_base.cc:1034] running on GCE node
W20260924 04:08:35.240029 3022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:08:35.240028 3025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:08:35.240232 3023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:08:35.240743 3017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:08:35.241421 3017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:08:35.242601 3017 hybrid_clock.cc:648] HybridClock initialized: now 1790222915242587 us; error 28 us; skew 500 ppm
Sep 24 04:08:35 dist-test-slave-00ld krb5kdc[2776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222915, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:08:35.245335 3017 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:08:35.246152 3017 webserver.cc:460] Webserver started at http://127.0.98.2:43597/ using document root <none> and password file <none>
I20260924 04:08:35.246906 3017 fs_manager.cc:359] Metadata directory not provided
I20260924 04:08:35.247005 3017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:08:35.247244 3017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:08:35.249375 3017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222898807720-392-0/minicluster-data/ts-1/data/instance:
uuid: "b11bb03ef1dc450781d09e9aff697f09"
format_stamp: "Formatted at 2026-09-24 04:08:35 on dist-test-slave-00ld"
server_key: "072194088a4388a15967afd6cb6e7a8c"
server_key_iv: "5262348dd762933dada472b7a4983a4c"
server_key_version: "encryptionkey@0"
I20260924 04:08:35.249954 3017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222898807720-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b11bb03ef1dc450781d09e9aff697f09"
format_stamp: "Formatted at 2026-09-24 04:08:35 on dist-test-slave-00ld"
server_key: "072194088a4388a15967afd6cb6e7a8c"
server_key_iv: "5262348dd762933dada472b7a4983a4c"
server_key_version: "encryptionkey@0"
I20260924 04:08:35.253648 3017 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:08:35.256188 3032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:35.257588 3017 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 04:08:35.257732 3017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222898807720-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222898807720-392-0/minicluster-data/ts-1/wal
uuid: "b11bb03ef1dc450781d09e9aff697f09"
format_stamp: "Formatted at 2026-09-24 04:08:35 on dist-test-slave-00ld"
server_key: "072194088a4388a15967afd6cb6e7a8c"
server_key_iv: "5262348dd762933dada472b7a4983a4c"
server_key_version: "encryptionkey@0"
I20260924 04:08:35.257856 3017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222898807720-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222898807720-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222898807720-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:08:35.271103 3017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:08:35.274308 3017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:08:35.274513 3017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:08:35.275148 3017 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:08:35.275955 3017 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:08:35.276033 3017 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:35.276106 3017 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:08:35.276139 3017 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:35.285390 3017 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43165
I20260924 04:08:35.285411 3145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43165 every 8 connection(s)
I20260924 04:08:35.286290 3017 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222898807720-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:08:35.294183 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 3017
I20260924 04:08:35.294329 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222898807720-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:08:35.294610 392 external_mini_cluster.cc:1428] Setting key 2d0bbe22a069a28b734d85fce14450a6
Sep 24 04:08:35 dist-test-slave-00ld krb5kdc[2776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222915, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:08:35.299588 2860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:35.288053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49575 (local address 127.0.98.62:37071)
0924 04:08:35.288183 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:35.288189 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:35.289016 (+ 827us) server_negotiation.cc:407] Connection header received
0924 04:08:35.290305 (+ 1289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:35.290308 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:35.290354 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:35.290427 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:35.291528 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:35.292073 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:35.293221 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:35.293439 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:35.293521 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:35.295974 (+ 2453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:35.295977 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:35.295979 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:35.296000 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:35.297464 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:35.297988 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:35.297993 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:35.297997 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:35.298099 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:35.298507 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:35.298512 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:35.298515 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:35.298732 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:35.298810 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:35.299301 (+ 491us) server_negotiation.cc:299] Negotiation successful
0924 04:08:35.299431 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 04:08:35.300527 3148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:35.288403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37071 (local address 127.0.98.2:49575)
0924 04:08:35.288876 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:35.288916 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:35.290123 (+ 1207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:35.290597 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:35.290603 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:35.290937 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:08:35.291391 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:35.291403 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:35.292174 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:35.292176 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:35.293113 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:35.293121 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:35.293572 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:35.293575 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:35.293707 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:35.294293 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:08:35.294334 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:08:35.295807 (+ 1473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:08:35.297594 (+ 1787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:35.297600 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:35.297615 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:35.297876 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:35.298216 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:35.298220 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:35.298222 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:35.298397 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:35.298821 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:08:35.298826 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:08:35.299051 (+ 225us) client_negotiation.cc:769] Sending connection context
0924 04:08:35.299233 (+ 182us) client_negotiation.cc:240] Negotiation successful
0924 04:08:35.299638 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":100,"threads_started":1}
I20260924 04:08:35.301765 3146 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37071
I20260924 04:08:35.302017 3146 heartbeater.cc:464] Registering TS with master...
I20260924 04:08:35.302488 3146 heartbeater.cc:511] Master 127.0.98.62:37071 requested a full tablet report, sending...
I20260924 04:08:35.303475 2821 ts_manager.cc:194] Registered new tserver with Master: b11bb03ef1dc450781d09e9aff697f09 (127.0.98.2:43165)
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 04:08:35.304028 2821 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49575
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222898807720-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222898807720-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:08:35.322115 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222898807720-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222898807720-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222898807720-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222898807720-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37071
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222898807720-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:08:35.445166 3151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:08:35.445463 3151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:08:35.445508 3151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:08:35.449622 3151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:08:35.449685 3151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:08:35.449747 3151 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:08:35.455142 3151 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222898807720-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222898807720-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222898807720-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222898807720-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3151
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222898807720-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:08:35.456151 3151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:08:35.457167 3151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:08:35.466641 3159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:08:35.466641 3156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:08:35.466636 3157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:08:35.466972 3151 server_base.cc:1034] running on GCE node
I20260924 04:08:35.467479 3151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:08:35.468217 3151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:08:35.469400 3151 hybrid_clock.cc:648] HybridClock initialized: now 1790222915469382 us; error 32 us; skew 500 ppm
Sep 24 04:08:35 dist-test-slave-00ld krb5kdc[2776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222915, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:08:35.472105 3151 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:08:35.473026 3151 webserver.cc:460] Webserver started at http://127.0.98.3:34519/ using document root <none> and password file <none>
I20260924 04:08:35.473587 3151 fs_manager.cc:359] Metadata directory not provided
I20260924 04:08:35.473677 3151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:08:35.473907 3151 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:08:35.476068 3151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222898807720-392-0/minicluster-data/ts-2/data/instance:
uuid: "7159d99d124c414a8d272e7344ae0b08"
format_stamp: "Formatted at 2026-09-24 04:08:35 on dist-test-slave-00ld"
server_key: "a272f4a82ac57ee27fd7ec946903d6e2"
server_key_iv: "ac12c4e233c15136ef96451e32540a38"
server_key_version: "encryptionkey@0"
I20260924 04:08:35.476740 3151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222898807720-392-0/minicluster-data/ts-2/wal/instance:
uuid: "7159d99d124c414a8d272e7344ae0b08"
format_stamp: "Formatted at 2026-09-24 04:08:35 on dist-test-slave-00ld"
server_key: "a272f4a82ac57ee27fd7ec946903d6e2"
server_key_iv: "ac12c4e233c15136ef96451e32540a38"
server_key_version: "encryptionkey@0"
I20260924 04:08:35.480341 3151 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 04:08:35.482853 3166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:35.484125 3151 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 04:08:35.484292 3151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222898807720-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222898807720-392-0/minicluster-data/ts-2/wal
uuid: "7159d99d124c414a8d272e7344ae0b08"
format_stamp: "Formatted at 2026-09-24 04:08:35 on dist-test-slave-00ld"
server_key: "a272f4a82ac57ee27fd7ec946903d6e2"
server_key_iv: "ac12c4e233c15136ef96451e32540a38"
server_key_version: "encryptionkey@0"
I20260924 04:08:35.484411 3151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222898807720-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222898807720-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222898807720-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:08:35.493889 3151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:08:35.496572 3151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:08:35.496806 3151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:08:35.497382 3151 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:08:35.498206 3151 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:08:35.498279 3151 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:35.498348 3151 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:08:35.498397 3151 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:35.507269 3151 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41143
I20260924 04:08:35.507287 3279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41143 every 8 connection(s)
I20260924 04:08:35.508159 3151 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222898807720-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:08:35.508385 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 3151
I20260924 04:08:35.508492 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790222898807720-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:08:35.508824 392 external_mini_cluster.cc:1428] Setting key 8858de8200ef54c855fdc6be4329fcc8
Sep 24 04:08:35 dist-test-slave-00ld krb5kdc[2776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222915, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:08:35.522311 2860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:35.510175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37117 (local address 127.0.98.62:37071)
0924 04:08:35.510309 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:35.510314 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:35.511250 (+ 936us) server_negotiation.cc:407] Connection header received
0924 04:08:35.512606 (+ 1356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:35.512608 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:35.512652 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:35.512772 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:35.513911 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:35.514447 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:35.515563 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:35.515771 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:35.515840 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:35.518703 (+ 2863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:35.518706 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:35.518708 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:35.518732 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:35.520190 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:35.520640 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:35.520643 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:35.520644 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:35.520774 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:35.521156 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:35.521160 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:35.521162 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:35.521413 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:35.521491 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:35.522037 (+ 546us) server_negotiation.cc:299] Negotiation successful
0924 04:08:35.522133 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 04:08:35.523370 3282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:35.510653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37071 (local address 127.0.98.3:37117)
0924 04:08:35.511098 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:35.511148 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:35.512418 (+ 1270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:35.512906 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:35.512913 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:35.513293 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:08:35.513782 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:35.513796 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:35.514562 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:35.514564 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:35.515455 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:35.515464 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:35.515888 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:35.515891 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:35.516061 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:35.516761 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:08:35.516781 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:08:35.518548 (+ 1767us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:08:35.520296 (+ 1748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:35.520300 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:35.520312 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:35.520549 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:35.520881 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:35.520884 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:35.520885 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:35.521028 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:35.521517 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:08:35.521524 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:08:35.521780 (+ 256us) client_negotiation.cc:769] Sending connection context
0924 04:08:35.521987 (+ 207us) client_negotiation.cc:240] Negotiation successful
0924 04:08:35.522376 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":122,"threads_started":1}
I20260924 04:08:35.524370 3280 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37071
I20260924 04:08:35.524623 3280 heartbeater.cc:464] Registering TS with master...
I20260924 04:08:35.525116 3280 heartbeater.cc:511] Master 127.0.98.62:37071 requested a full tablet report, sending...
I20260924 04:08:35.526104 2821 ts_manager.cc:194] Registered new tserver with Master: 7159d99d124c414a8d272e7344ae0b08 (127.0.98.3:41143)
I20260924 04:08:35.526681 2821 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:37117
I20260924 04:08:35.534610 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 04:08:35.547294 2860 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:08:35.545840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59366 (local address 127.0.98.62:37071)
0924 04:08:35.545986 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:35.545991 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:35.546181 (+ 190us) server_negotiation.cc:407] Connection header received
0924 04:08:35.546291 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:35.546293 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:35.546337 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:35.546408 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:35.547147 (+ 739us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59366: BlockingRecv error: recv got EOF from 127.0.0.1:59366 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 04:08:35.548823 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 2883
I20260924 04:08:35.554114 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 3017
I20260924 04:08:35.559453 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 3151
I20260924 04:08:35.564735 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 2789
2026-09-24T04:08:35Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2100 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790222898807720-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:08:35 dist-test-slave-00ld krb5kdc[3292](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:08:35 dist-test-slave-00ld krb5kdc[3292](info): set up 2 sockets
Sep 24 04:08:35 dist-test-slave-00ld krb5kdc[3292](info): commencing operation
krb5kdc: starting...
W20260924 04:08:36.596614 392 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790222898807720-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790222898807720-392-0/krb5kdc/test-user.keytab.
Sep 24 04:08:36 dist-test-slave-00ld krb5kdc[3292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:08:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:08:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:08:36.719726 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790222898807720-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790222898807720-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790222898807720-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37627
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37627
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:08:36.828783 3305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:08:36.829093 3305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:08:36.829169 3305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:08:36.833195 3305 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:08:36.833293 3305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:08:36.833330 3305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:08:36.833391 3305 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:08:36.833429 3305 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:08:36.838783 3305 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790222898807720-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790222898807720-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37627
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37627
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3305
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790222898807720-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:08:36.839886 3305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:08:36.840945 3305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:08:36.849248 3310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:08:36.849248 3313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:08:36.849324 3305 server_base.cc:1034] running on GCE node
W20260924 04:08:36.849287 3311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:08:36.850155 3305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:08:36.851274 3305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:08:36.852475 3305 hybrid_clock.cc:648] HybridClock initialized: now 1790222916852444 us; error 53 us; skew 500 ppm
Sep 24 04:08:36 dist-test-slave-00ld krb5kdc[3292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222916, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:08:36.859751 3305 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:08:36.860762 3305 webserver.cc:460] Webserver started at http://127.0.98.62:33603/ using document root <none> and password file <none>
I20260924 04:08:36.861395 3305 fs_manager.cc:359] Metadata directory not provided
I20260924 04:08:36.861490 3305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:08:36.861728 3305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:08:36.863809 3305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790222898807720-392-0/minicluster-data/master-0/data/instance:
uuid: "974ce9a05efa4aa0903eb0aba72e667a"
format_stamp: "Formatted at 2026-09-24 04:08:36 on dist-test-slave-00ld"
server_key: "286e68c8ba24cfc5673dd9a8063df3a9"
server_key_iv: "8af0c4dfde377cc1f56e76384be92f60"
server_key_version: "encryptionkey@0"
I20260924 04:08:36.864392 3305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790222898807720-392-0/minicluster-data/master-0/wal/instance:
uuid: "974ce9a05efa4aa0903eb0aba72e667a"
format_stamp: "Formatted at 2026-09-24 04:08:36 on dist-test-slave-00ld"
server_key: "286e68c8ba24cfc5673dd9a8063df3a9"
server_key_iv: "8af0c4dfde377cc1f56e76384be92f60"
server_key_version: "encryptionkey@0"
I20260924 04:08:36.868129 3305 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 04:08:36.870639 3320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:36.871858 3305 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:08:36.871999 3305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790222898807720-392-0/minicluster-data/master-0/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790222898807720-392-0/minicluster-data/master-0/wal
uuid: "974ce9a05efa4aa0903eb0aba72e667a"
format_stamp: "Formatted at 2026-09-24 04:08:36 on dist-test-slave-00ld"
server_key: "286e68c8ba24cfc5673dd9a8063df3a9"
server_key_iv: "8af0c4dfde377cc1f56e76384be92f60"
server_key_version: "encryptionkey@0"
I20260924 04:08:36.872108 3305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790222898807720-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790222898807720-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790222898807720-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:08:36.887814 3305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:08:36.891170 3305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:08:36.891388 3305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:08:36.898067 3305 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37627
I20260924 04:08:36.898339 3372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37627 every 8 connection(s)
I20260924 04:08:36.899032 3305 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:08:36.901655 3373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:08:36.905906 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 3305
I20260924 04:08:36.906051 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790222898807720-392-0/minicluster-data/master-0/wal/instance
I20260924 04:08:36.906318 392 external_mini_cluster.cc:1428] Setting key 024442e2900ee5ef4d17f3822c17d983
I20260924 04:08:36.907140 3373 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 974ce9a05efa4aa0903eb0aba72e667a: Bootstrap starting.
I20260924 04:08:36.909622 3373 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 974ce9a05efa4aa0903eb0aba72e667a: Neither blocks nor log segments found. Creating new log.
I20260924 04:08:36.910576 3373 log.cc:824] T 00000000000000000000000000000000 P 974ce9a05efa4aa0903eb0aba72e667a: Log is configured to *not* fsync() on all Append() calls
I20260924 04:08:36.912563 3373 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 974ce9a05efa4aa0903eb0aba72e667a: No bootstrap required, opened a new log
Sep 24 04:08:36 dist-test-slave-00ld krb5kdc[3292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:08:36.916304 3373 raft_consensus.cc:348] T 00000000000000000000000000000000 P 974ce9a05efa4aa0903eb0aba72e667a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "974ce9a05efa4aa0903eb0aba72e667a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37627 } }
I20260924 04:08:36.916592 3373 raft_consensus.cc:374] T 00000000000000000000000000000000 P 974ce9a05efa4aa0903eb0aba72e667a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:08:36.916661 3373 raft_consensus.cc:729] T 00000000000000000000000000000000 P 974ce9a05efa4aa0903eb0aba72e667a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 974ce9a05efa4aa0903eb0aba72e667a, State: Initialized, Role: FOLLOWER
I20260924 04:08:36.917173 3373 consensus_queue.cc:260] T 00000000000000000000000000000000 P 974ce9a05efa4aa0903eb0aba72e667a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "974ce9a05efa4aa0903eb0aba72e667a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37627 } }
I20260924 04:08:36.917299 3373 raft_consensus.cc:388] T 00000000000000000000000000000000 P 974ce9a05efa4aa0903eb0aba72e667a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:08:36.917384 3373 raft_consensus.cc:482] T 00000000000000000000000000000000 P 974ce9a05efa4aa0903eb0aba72e667a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:08:36.917482 3373 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 974ce9a05efa4aa0903eb0aba72e667a [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:08:36.918341 3373 raft_consensus.cc:504] T 00000000000000000000000000000000 P 974ce9a05efa4aa0903eb0aba72e667a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "974ce9a05efa4aa0903eb0aba72e667a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37627 } }
I20260924 04:08:36.918684 3373 leader_election.cc:302] T 00000000000000000000000000000000 P 974ce9a05efa4aa0903eb0aba72e667a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 974ce9a05efa4aa0903eb0aba72e667a; no voters:
I20260924 04:08:36.919020 3373 leader_election.cc:288] T 00000000000000000000000000000000 P 974ce9a05efa4aa0903eb0aba72e667a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:08:36.919090 3378 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 974ce9a05efa4aa0903eb0aba72e667a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:08:36.919293 3378 raft_consensus.cc:686] T 00000000000000000000000000000000 P 974ce9a05efa4aa0903eb0aba72e667a [term 1 LEADER]: Becoming Leader. State: Replica: 974ce9a05efa4aa0903eb0aba72e667a, State: Running, Role: LEADER
I20260924 04:08:36.919658 3378 consensus_queue.cc:237] T 00000000000000000000000000000000 P 974ce9a05efa4aa0903eb0aba72e667a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "974ce9a05efa4aa0903eb0aba72e667a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37627 } }
I20260924 04:08:36.920300 3373 sys_catalog.cc:564] T 00000000000000000000000000000000 P 974ce9a05efa4aa0903eb0aba72e667a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:08:36.921355 3380 sys_catalog.cc:455] T 00000000000000000000000000000000 P 974ce9a05efa4aa0903eb0aba72e667a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "974ce9a05efa4aa0903eb0aba72e667a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "974ce9a05efa4aa0903eb0aba72e667a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37627 } } }
I20260924 04:08:36.921486 3380 sys_catalog.cc:458] T 00000000000000000000000000000000 P 974ce9a05efa4aa0903eb0aba72e667a [sys.catalog]: This master's current role is: LEADER
I20260924 04:08:36.921617 3379 sys_catalog.cc:455] T 00000000000000000000000000000000 P 974ce9a05efa4aa0903eb0aba72e667a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 974ce9a05efa4aa0903eb0aba72e667a. Latest consensus state: current_term: 1 leader_uuid: "974ce9a05efa4aa0903eb0aba72e667a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "974ce9a05efa4aa0903eb0aba72e667a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37627 } } }
I20260924 04:08:36.921746 3376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:36.907625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41100 (local address 127.0.98.62:37627)
0924 04:08:36.908025 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:36.908042 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:36.908088 (+ 46us) server_negotiation.cc:407] Connection header received
0924 04:08:36.908901 (+ 813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:36.908923 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:36.909308 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:36.909672 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:36.910434 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:36.911747 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:36.912636 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:36.913338 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:36.913514 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:36.915936 (+ 2422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:36.915942 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:36.915957 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:36.915987 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:36.918019 (+ 2032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:36.918466 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:36.918473 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:36.918479 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:36.918615 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:36.918924 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:36.918927 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:36.918929 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:36.919453 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:36.920218 (+ 765us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:36.920339 (+ 121us) server_negotiation.cc:299] Negotiation successful
0924 04:08:36.920653 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":107,"threads_started":1}
I20260924 04:08:36.922063 3387 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:08:36.921792 3379 sys_catalog.cc:458] T 00000000000000000000000000000000 P 974ce9a05efa4aa0903eb0aba72e667a [sys.catalog]: This master's current role is: LEADER
I20260924 04:08:36.924883 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:08:36.925115 3387 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:08:36.931509 3387 catalog_manager.cc:1347] Generated new cluster ID: 1f8635e6403a42a5808a49365f7fdd25
I20260924 04:08:36.931578 3387 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:08:36.940068 3387 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:08:36.941138 3387 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:08:36.949172 3376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:36.943381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41102 (local address 127.0.98.62:37627)
0924 04:08:36.943485 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:36.943489 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:36.943830 (+ 341us) server_negotiation.cc:407] Connection header received
0924 04:08:36.944088 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:36.944091 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:36.944135 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:36.944216 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:36.944827 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:36.945580 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:36.946800 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:36.947143 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:36.947227 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:36.948145 (+ 918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:36.948149 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:36.948152 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:08:36.948718 (+ 566us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:36.948749 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:08:36.948767 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:36.948847 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:36.948978 (+ 131us) server_negotiation.cc:299] Negotiation successful
0924 04:08:36.949034 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 04:08:36.954483 3387 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 974ce9a05efa4aa0903eb0aba72e667a: Generated new TSK 0
I20260924 04:08:36.955336 3387 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:08:36.964353 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 3305
2026-09-24T04:08:36Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1397 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222898807720-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:08:37 dist-test-slave-00ld krb5kdc[3406](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:08:37 dist-test-slave-00ld krb5kdc[3406](info): set up 2 sockets
Sep 24 04:08:37 dist-test-slave-00ld krb5kdc[3406](info): commencing operation
krb5kdc: starting...
W20260924 04:08:38.040647 392 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222898807720-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222898807720-392-0/krb5kdc/test-user.keytab.
Sep 24 04:08:38 dist-test-slave-00ld krb5kdc[3406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222918, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:08:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:08:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:08:38.160271 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222898807720-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222898807720-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222898807720-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38079
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38079
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:08:38.266824 3419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:08:38.267158 3419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:08:38.267246 3419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:08:38.271242 3419 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:08:38.271340 3419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:08:38.271378 3419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:08:38.271420 3419 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:08:38.271451 3419 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:08:38.276736 3419 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222898807720-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222898807720-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38079
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38079
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3419
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222898807720-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:08:38.277832 3419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:08:38.278865 3419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:08:38.287362 3424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:08:38.287343 3425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:08:38.287670 3419 server_base.cc:1034] running on GCE node
W20260924 04:08:38.287364 3427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:08:38.288204 3419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:08:38.289324 3419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:08:38.290520 3419 hybrid_clock.cc:648] HybridClock initialized: now 1790222918290501 us; error 32 us; skew 500 ppm
Sep 24 04:08:38 dist-test-slave-00ld krb5kdc[3406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222918, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:08:38.293157 3419 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:08:38.294261 3419 webserver.cc:460] Webserver started at http://127.0.98.62:40237/ using document root <none> and password file <none>
I20260924 04:08:38.295056 3419 fs_manager.cc:359] Metadata directory not provided
I20260924 04:08:38.295149 3419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:08:38.295423 3419 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:08:38.298413 3419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222898807720-392-0/minicluster-data/master-0/data/instance:
uuid: "2801def25cbb4827b79cc8831908c795"
format_stamp: "Formatted at 2026-09-24 04:08:38 on dist-test-slave-00ld"
server_key: "949a1b229794f26003fef0fda15ba061"
server_key_iv: "d822c56dab09ef28b9bc6197590783bd"
server_key_version: "encryptionkey@0"
I20260924 04:08:38.299131 3419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222898807720-392-0/minicluster-data/master-0/wal/instance:
uuid: "2801def25cbb4827b79cc8831908c795"
format_stamp: "Formatted at 2026-09-24 04:08:38 on dist-test-slave-00ld"
server_key: "949a1b229794f26003fef0fda15ba061"
server_key_iv: "d822c56dab09ef28b9bc6197590783bd"
server_key_version: "encryptionkey@0"
I20260924 04:08:38.303712 3419 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 04:08:38.305994 3434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:38.307152 3419 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:08:38.307255 3419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222898807720-392-0/minicluster-data/master-0/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222898807720-392-0/minicluster-data/master-0/wal
uuid: "2801def25cbb4827b79cc8831908c795"
format_stamp: "Formatted at 2026-09-24 04:08:38 on dist-test-slave-00ld"
server_key: "949a1b229794f26003fef0fda15ba061"
server_key_iv: "d822c56dab09ef28b9bc6197590783bd"
server_key_version: "encryptionkey@0"
I20260924 04:08:38.307335 3419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222898807720-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222898807720-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222898807720-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:08:38.330070 3419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:08:38.333011 3419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:08:38.333174 3419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:08:38.340273 3419 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38079
I20260924 04:08:38.340579 3486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38079 every 8 connection(s)
I20260924 04:08:38.341338 3419 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:08:38.344182 3487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:08:38.346429 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 3419
I20260924 04:08:38.346556 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222898807720-392-0/minicluster-data/master-0/wal/instance
I20260924 04:08:38.346849 392 external_mini_cluster.cc:1428] Setting key beb03108bdbed84a29d4dad78b718a4b
I20260924 04:08:38.350517 3487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2801def25cbb4827b79cc8831908c795: Bootstrap starting.
I20260924 04:08:38.352802 3487 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2801def25cbb4827b79cc8831908c795: Neither blocks nor log segments found. Creating new log.
I20260924 04:08:38.353542 3487 log.cc:824] T 00000000000000000000000000000000 P 2801def25cbb4827b79cc8831908c795: Log is configured to *not* fsync() on all Append() calls
I20260924 04:08:38.355468 3487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2801def25cbb4827b79cc8831908c795: No bootstrap required, opened a new log
Sep 24 04:08:38 dist-test-slave-00ld krb5kdc[3406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222918, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:08:38.358557 3487 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2801def25cbb4827b79cc8831908c795 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2801def25cbb4827b79cc8831908c795" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38079 } }
I20260924 04:08:38.358826 3487 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2801def25cbb4827b79cc8831908c795 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:08:38.358912 3487 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2801def25cbb4827b79cc8831908c795 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2801def25cbb4827b79cc8831908c795, State: Initialized, Role: FOLLOWER
I20260924 04:08:38.359359 3487 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2801def25cbb4827b79cc8831908c795 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2801def25cbb4827b79cc8831908c795" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38079 } }
I20260924 04:08:38.359522 3487 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2801def25cbb4827b79cc8831908c795 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:08:38.359604 3487 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2801def25cbb4827b79cc8831908c795 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:08:38.359707 3487 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2801def25cbb4827b79cc8831908c795 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:08:38.360608 3487 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2801def25cbb4827b79cc8831908c795 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2801def25cbb4827b79cc8831908c795" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38079 } }
I20260924 04:08:38.360985 3487 leader_election.cc:302] T 00000000000000000000000000000000 P 2801def25cbb4827b79cc8831908c795 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2801def25cbb4827b79cc8831908c795; no voters:
I20260924 04:08:38.361331 3487 leader_election.cc:288] T 00000000000000000000000000000000 P 2801def25cbb4827b79cc8831908c795 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:08:38.361416 3492 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2801def25cbb4827b79cc8831908c795 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:08:38.361586 3492 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2801def25cbb4827b79cc8831908c795 [term 1 LEADER]: Becoming Leader. State: Replica: 2801def25cbb4827b79cc8831908c795, State: Running, Role: LEADER
I20260924 04:08:38.361946 3492 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2801def25cbb4827b79cc8831908c795 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2801def25cbb4827b79cc8831908c795" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38079 } }
I20260924 04:08:38.362358 3490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:38.348251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37932 (local address 127.0.98.62:38079)
0924 04:08:38.348836 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:38.348854 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:38.348893 (+ 39us) server_negotiation.cc:407] Connection header received
0924 04:08:38.349721 (+ 828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:38.349742 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:38.350116 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:38.350471 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:38.351290 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:38.352296 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:38.353307 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:38.353841 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:38.354009 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:38.356556 (+ 2547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:38.356567 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:38.356581 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:38.356610 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:38.359203 (+ 2593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:38.359659 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:38.359665 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:38.359671 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:38.359836 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:38.360111 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:38.360116 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:38.360118 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:38.360532 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:38.360794 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:38.361019 (+ 225us) server_negotiation.cc:299] Negotiation successful
0924 04:08:38.361350 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":139,"threads_started":1}
I20260924 04:08:38.362637 3487 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2801def25cbb4827b79cc8831908c795 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:08:38.363579 3493 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2801def25cbb4827b79cc8831908c795 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2801def25cbb4827b79cc8831908c795. Latest consensus state: current_term: 1 leader_uuid: "2801def25cbb4827b79cc8831908c795" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2801def25cbb4827b79cc8831908c795" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38079 } } }
I20260924 04:08:38.363709 3493 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2801def25cbb4827b79cc8831908c795 [sys.catalog]: This master's current role is: LEADER
I20260924 04:08:38.364039 3494 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2801def25cbb4827b79cc8831908c795 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2801def25cbb4827b79cc8831908c795" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2801def25cbb4827b79cc8831908c795" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38079 } } }
I20260924 04:08:38.364114 3494 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2801def25cbb4827b79cc8831908c795 [sys.catalog]: This master's current role is: LEADER
W20260924 04:08:38.367923 3507 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2801def25cbb4827b79cc8831908c795: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:08:38.368005 3507 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:08:38.368067 3501 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:08:38.368956 3501 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:08:38.374640 3501 catalog_manager.cc:1347] Generated new cluster ID: 1c7209f0156e4e3cae016f7677893dd5
I20260924 04:08:38.374714 3501 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:08:38.391084 3501 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:08:38.392019 3501 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:08:38.396277 3501 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2801def25cbb4827b79cc8831908c795: Generated new TSK 0
I20260924 04:08:38.397106 3501 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222898807720-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222898807720-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:08:38.431840 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222898807720-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222898807720-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222898807720-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222898807720-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38079
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222898807720-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:08:38.545392 3513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:08:38.545665 3513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:08:38.545713 3513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:08:38.549952 3513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:08:38.550019 3513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:08:38.550112 3513 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:08:38.555487 3513 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222898807720-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222898807720-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222898807720-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222898807720-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38079
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3513
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222898807720-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:08:38.556607 3513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:08:38.557791 3513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:08:38.567639 3521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:08:38.567760 3513 server_base.cc:1034] running on GCE node
W20260924 04:08:38.567610 3519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:08:38.568019 3518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:08:38.568445 3513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:08:38.569151 3513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:08:38.570362 3513 hybrid_clock.cc:648] HybridClock initialized: now 1790222918570341 us; error 48 us; skew 500 ppm
Sep 24 04:08:38 dist-test-slave-00ld krb5kdc[3406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222918, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:08:38.573124 3513 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:08:38.574062 3513 webserver.cc:460] Webserver started at http://127.0.98.1:38617/ using document root <none> and password file <none>
I20260924 04:08:38.574615 3513 fs_manager.cc:359] Metadata directory not provided
I20260924 04:08:38.574703 3513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:08:38.574961 3513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:08:38.577040 3513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222898807720-392-0/minicluster-data/ts-0/data/instance:
uuid: "2245aed142744375b86f1662d3a76202"
format_stamp: "Formatted at 2026-09-24 04:08:38 on dist-test-slave-00ld"
server_key: "becef09015462546cb466876fecc2023"
server_key_iv: "48aa224918bbb10dc9eed5d152b03a9a"
server_key_version: "encryptionkey@0"
I20260924 04:08:38.577651 3513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222898807720-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2245aed142744375b86f1662d3a76202"
format_stamp: "Formatted at 2026-09-24 04:08:38 on dist-test-slave-00ld"
server_key: "becef09015462546cb466876fecc2023"
server_key_iv: "48aa224918bbb10dc9eed5d152b03a9a"
server_key_version: "encryptionkey@0"
I20260924 04:08:38.581271 3513 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:08:38.583604 3528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:38.585035 3513 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:08:38.585162 3513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222898807720-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222898807720-392-0/minicluster-data/ts-0/wal
uuid: "2245aed142744375b86f1662d3a76202"
format_stamp: "Formatted at 2026-09-24 04:08:38 on dist-test-slave-00ld"
server_key: "becef09015462546cb466876fecc2023"
server_key_iv: "48aa224918bbb10dc9eed5d152b03a9a"
server_key_version: "encryptionkey@0"
I20260924 04:08:38.585248 3513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222898807720-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222898807720-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222898807720-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:08:38.595633 3513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:08:38.598397 3513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:08:38.598538 3513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:08:38.599025 3513 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:08:38.599740 3513 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:08:38.599785 3513 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:38.599822 3513 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:08:38.599843 3513 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:38.609246 3513 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38719
I20260924 04:08:38.609302 3641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38719 every 8 connection(s)
I20260924 04:08:38.610209 3513 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222898807720-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:08:38.618327 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 3513
I20260924 04:08:38.618505 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222898807720-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:08:38.618855 392 external_mini_cluster.cc:1428] Setting key 94e4daba3f6c0f6ce16c425cd4e60a09
Sep 24 04:08:38 dist-test-slave-00ld krb5kdc[3406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222918, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:08:38.625089 3490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:38.612146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36843 (local address 127.0.98.62:38079)
0924 04:08:38.612277 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:38.612280 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:38.613315 (+ 1035us) server_negotiation.cc:407] Connection header received
0924 04:08:38.614765 (+ 1450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:38.614768 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:38.614821 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:38.614904 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:38.616097 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:38.616653 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:38.617879 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:38.618192 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:38.618325 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:38.621360 (+ 3035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:38.621367 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:38.621373 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:38.621406 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:38.622958 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:38.623506 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:38.623512 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:38.623516 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:38.623598 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:38.623973 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:38.623979 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:38.623983 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:38.624162 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:38.624270 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:38.624804 (+ 534us) server_negotiation.cc:299] Negotiation successful
0924 04:08:38.624924 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 04:08:38.625960 3644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:38.612573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38079 (local address 127.0.98.1:36843)
0924 04:08:38.613140 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:38.613183 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:38.614540 (+ 1357us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:38.615076 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:38.615083 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:38.615449 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:08:38.615949 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:38.615962 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:38.616837 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:38.616840 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:38.617752 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:38.617761 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:38.618446 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:38.618451 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:38.618667 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:38.619352 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:08:38.619369 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:08:38.621182 (+ 1813us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:08:38.623112 (+ 1930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:38.623118 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:38.623133 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:38.623398 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:38.623711 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:38.623714 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:38.623716 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:38.623869 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:38.624268 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:08:38.624275 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:08:38.624529 (+ 254us) client_negotiation.cc:769] Sending connection context
0924 04:08:38.624764 (+ 235us) client_negotiation.cc:240] Negotiation successful
0924 04:08:38.625121 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":121,"threads_started":1}
I20260924 04:08:38.627624 3642 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38079
I20260924 04:08:38.627871 3642 heartbeater.cc:464] Registering TS with master...
I20260924 04:08:38.628348 3642 heartbeater.cc:511] Master 127.0.98.62:38079 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 04:08:38.629899 3451 ts_manager.cc:194] Registered new tserver with Master: 2245aed142744375b86f1662d3a76202 (127.0.98.1:38719)
I20260924 04:08:38.631130 3451 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:36843
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222898807720-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222898807720-392-0/krb5kdc/kudu_127.0.98.2.keytab.
W20260924 04:08:38.799875 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.2: real 0.179s user 0.000s sys 0.003s
I20260924 04:08:38.800432 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222898807720-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222898807720-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222898807720-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222898807720-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38079
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222898807720-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:08:38.910754 3647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:08:38.911070 3647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:08:38.911114 3647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:08:38.915165 3647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:08:38.915230 3647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:08:38.915320 3647 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:08:38.920594 3647 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222898807720-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222898807720-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222898807720-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222898807720-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38079
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3647
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222898807720-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:08:38.921686 3647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:08:38.922734 3647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:08:38.932513 3652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:08:38.932571 3653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:08:38.932629 3647 server_base.cc:1034] running on GCE node
W20260924 04:08:38.932579 3655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:08:38.933286 3647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:08:38.933948 3647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:08:38.935128 3647 hybrid_clock.cc:648] HybridClock initialized: now 1790222918935110 us; error 37 us; skew 500 ppm
Sep 24 04:08:38 dist-test-slave-00ld krb5kdc[3406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222918, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:08:38.938390 3647 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:08:38.939278 3647 webserver.cc:460] Webserver started at http://127.0.98.2:45963/ using document root <none> and password file <none>
I20260924 04:08:38.939848 3647 fs_manager.cc:359] Metadata directory not provided
I20260924 04:08:38.939939 3647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:08:38.940176 3647 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:08:38.942315 3647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222898807720-392-0/minicluster-data/ts-1/data/instance:
uuid: "67ae8287f44e4a1585dbf04936e41864"
format_stamp: "Formatted at 2026-09-24 04:08:38 on dist-test-slave-00ld"
server_key: "185499adb0bfad140c352f640301114f"
server_key_iv: "0ffda932e1a07dd29bc16fc03d1462f2"
server_key_version: "encryptionkey@0"
I20260924 04:08:38.942879 3647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222898807720-392-0/minicluster-data/ts-1/wal/instance:
uuid: "67ae8287f44e4a1585dbf04936e41864"
format_stamp: "Formatted at 2026-09-24 04:08:38 on dist-test-slave-00ld"
server_key: "185499adb0bfad140c352f640301114f"
server_key_iv: "0ffda932e1a07dd29bc16fc03d1462f2"
server_key_version: "encryptionkey@0"
I20260924 04:08:38.946643 3647 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.001s
I20260924 04:08:38.949096 3662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:38.950338 3647 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:08:38.950485 3647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222898807720-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222898807720-392-0/minicluster-data/ts-1/wal
uuid: "67ae8287f44e4a1585dbf04936e41864"
format_stamp: "Formatted at 2026-09-24 04:08:38 on dist-test-slave-00ld"
server_key: "185499adb0bfad140c352f640301114f"
server_key_iv: "0ffda932e1a07dd29bc16fc03d1462f2"
server_key_version: "encryptionkey@0"
I20260924 04:08:38.950599 3647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222898807720-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222898807720-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222898807720-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:08:38.973624 3647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:08:38.977217 3647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:08:38.977429 3647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:08:38.978044 3647 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:08:38.978894 3647 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:08:38.978967 3647 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:38.979040 3647 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:08:38.979091 3647 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:38.988014 3775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42801 every 8 connection(s)
I20260924 04:08:38.988025 3647 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42801
I20260924 04:08:38.989192 3647 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222898807720-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:08:38.996954 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 3647
I20260924 04:08:38.997098 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222898807720-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:08:38.997380 392 external_mini_cluster.cc:1428] Setting key 327eb3879a95873e261f054e292b3b65
Sep 24 04:08:38 dist-test-slave-00ld krb5kdc[3406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222918, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:08:39.003791 3490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:38.991216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48293 (local address 127.0.98.62:38079)
0924 04:08:38.991354 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:38.991358 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:38.992258 (+ 900us) server_negotiation.cc:407] Connection header received
0924 04:08:38.993735 (+ 1477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:38.993739 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:38.993798 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:38.993921 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:38.995148 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:38.995693 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:38.996905 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:38.997184 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:38.997285 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:39.000134 (+ 2849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:39.000137 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:39.000140 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:39.000174 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:39.001680 (+ 1506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:39.002234 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:39.002240 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:39.002242 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:39.002324 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:39.002668 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:39.002673 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:39.002675 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:39.002862 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:39.002970 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:39.003489 (+ 519us) server_negotiation.cc:299] Negotiation successful
0924 04:08:39.003610 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 04:08:39.004658 3778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:38.991616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38079 (local address 127.0.98.2:48293)
0924 04:08:38.992086 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:38.992130 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:38.993500 (+ 1370us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:38.994105 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:38.994112 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:38.994496 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:08:38.994989 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:38.995003 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:38.995823 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:38.995826 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:38.996773 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:38.996782 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:38.997345 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:38.997350 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:38.997563 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:38.998297 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:08:38.998318 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:08:38.999966 (+ 1648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:08:39.001849 (+ 1883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:39.001855 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:39.001872 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:39.002128 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:39.002425 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:39.002427 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:39.002430 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:39.002571 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:39.002975 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:08:39.002985 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:08:39.003242 (+ 257us) client_negotiation.cc:769] Sending connection context
0924 04:08:39.003449 (+ 207us) client_negotiation.cc:240] Negotiation successful
0924 04:08:39.003788 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":110,"threads_started":1}
I20260924 04:08:39.005975 3776 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38079
I20260924 04:08:39.006253 3776 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 04:08:39.006835 3776 heartbeater.cc:511] Master 127.0.98.62:38079 requested a full tablet report, sending...
I20260924 04:08:39.007968 3451 ts_manager.cc:194] Registered new tserver with Master: 67ae8287f44e4a1585dbf04936e41864 (127.0.98.2:42801)
I20260924 04:08:39.008671 3451 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:48293
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222898807720-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222898807720-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:08:39.025851 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222898807720-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222898807720-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222898807720-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222898807720-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38079
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222898807720-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:08:39.136215 3781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:08:39.136466 3781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:08:39.136508 3781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:08:39.140561 3781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:08:39.140627 3781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:08:39.140739 3781 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:08:39.146128 3781 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222898807720-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222898807720-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222898807720-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222898807720-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38079
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3781
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222898807720-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:08:39.147105 3781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:08:39.148075 3781 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:08:39.157968 3787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:08:39.157945 3786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:08:39.157964 3789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:08:39.158746 3781 server_base.cc:1034] running on GCE node
I20260924 04:08:39.159164 3781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:08:39.159797 3781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:08:39.160988 3781 hybrid_clock.cc:648] HybridClock initialized: now 1790222919160973 us; error 28 us; skew 500 ppm
Sep 24 04:08:39 dist-test-slave-00ld krb5kdc[3406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222919, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:08:39.163766 3781 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:08:39.164773 3781 webserver.cc:460] Webserver started at http://127.0.98.3:39159/ using document root <none> and password file <none>
I20260924 04:08:39.165330 3781 fs_manager.cc:359] Metadata directory not provided
I20260924 04:08:39.165428 3781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:08:39.165637 3781 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:08:39.167623 3781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222898807720-392-0/minicluster-data/ts-2/data/instance:
uuid: "9c262bfa161d4fd495910dfaf33f6e72"
format_stamp: "Formatted at 2026-09-24 04:08:39 on dist-test-slave-00ld"
server_key: "a87d0fc303aa45dd491110908dac7113"
server_key_iv: "4d05e1f076775f2070ffab4699e34fa7"
server_key_version: "encryptionkey@0"
I20260924 04:08:39.168184 3781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222898807720-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9c262bfa161d4fd495910dfaf33f6e72"
format_stamp: "Formatted at 2026-09-24 04:08:39 on dist-test-slave-00ld"
server_key: "a87d0fc303aa45dd491110908dac7113"
server_key_iv: "4d05e1f076775f2070ffab4699e34fa7"
server_key_version: "encryptionkey@0"
I20260924 04:08:39.172019 3781 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260924 04:08:39.174652 3796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:39.175938 3781 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 04:08:39.176072 3781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222898807720-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222898807720-392-0/minicluster-data/ts-2/wal
uuid: "9c262bfa161d4fd495910dfaf33f6e72"
format_stamp: "Formatted at 2026-09-24 04:08:39 on dist-test-slave-00ld"
server_key: "a87d0fc303aa45dd491110908dac7113"
server_key_iv: "4d05e1f076775f2070ffab4699e34fa7"
server_key_version: "encryptionkey@0"
I20260924 04:08:39.176187 3781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222898807720-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222898807720-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222898807720-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:08:39.195016 3781 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:08:39.198074 3781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:08:39.198284 3781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:08:39.198875 3781 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:08:39.199703 3781 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:08:39.199774 3781 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:39.199846 3781 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:08:39.199899 3781 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:39.209049 3781 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43483
I20260924 04:08:39.209066 3909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43483 every 8 connection(s)
I20260924 04:08:39.209975 3781 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222898807720-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:08:39.212256 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 3781
I20260924 04:08:39.212390 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790222898807720-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:08:39.212661 392 external_mini_cluster.cc:1428] Setting key 825725e929806ff7633b3abaa7865b39
Sep 24 04:08:39 dist-test-slave-00ld krb5kdc[3406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222919, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:08:39.223745 3490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:39.211822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37195 (local address 127.0.98.62:38079)
0924 04:08:39.212003 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:39.212008 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:39.213042 (+ 1034us) server_negotiation.cc:407] Connection header received
0924 04:08:39.214858 (+ 1816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:39.214861 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:39.214904 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:39.215012 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:39.216179 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:39.216796 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:39.217910 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:39.218115 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:39.218191 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:39.220716 (+ 2525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:39.220723 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:39.220728 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:39.220758 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:39.222029 (+ 1271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:39.222489 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:39.222491 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:39.222492 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:39.222559 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:39.222835 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:39.222838 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:39.222839 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:39.222998 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:39.223071 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:39.223532 (+ 461us) server_negotiation.cc:299] Negotiation successful
0924 04:08:39.223620 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260924 04:08:39.224571 3912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:39.212341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38079 (local address 127.0.98.3:37195)
0924 04:08:39.212882 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:39.212937 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:39.214625 (+ 1688us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:39.215199 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:39.215208 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:39.215590 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:08:39.216060 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:39.216072 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:39.216896 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:39.216898 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:39.217804 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:39.217814 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:39.218237 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:39.218240 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:39.218397 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:39.219045 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:08:39.219062 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:08:39.220530 (+ 1468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:08:39.222127 (+ 1597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:39.222132 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:39.222144 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:39.222397 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:39.222633 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:39.222635 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:39.222636 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:39.222760 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:39.223078 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:08:39.223083 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:08:39.223333 (+ 250us) client_negotiation.cc:769] Sending connection context
0924 04:08:39.223488 (+ 155us) client_negotiation.cc:240] Negotiation successful
0924 04:08:39.223730 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":138,"threads_started":1}
I20260924 04:08:39.225852 3910 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38079
I20260924 04:08:39.226140 3910 heartbeater.cc:464] Registering TS with master...
I20260924 04:08:39.226714 3910 heartbeater.cc:511] Master 127.0.98.62:38079 requested a full tablet report, sending...
I20260924 04:08:39.227746 3451 ts_manager.cc:194] Registered new tserver with Master: 9c262bfa161d4fd495910dfaf33f6e72 (127.0.98.3:43483)
I20260924 04:08:39.228358 3451 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:37195
I20260924 04:08:39.238468 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 04:08:39 dist-test-slave-00ld krb5kdc[3406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222919, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 24 04:08:39 dist-test-slave-00ld krb5kdc[3406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222919, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:08:39.263238 3490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:39.254994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37940 (local address 127.0.98.62:38079)
0924 04:08:39.255142 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:39.255146 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:39.255236 (+ 90us) server_negotiation.cc:407] Connection header received
0924 04:08:39.255365 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:39.255368 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:39.255410 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:39.255479 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:39.256150 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:39.256656 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:39.257509 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:39.257718 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:39.257792 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:39.260222 (+ 2430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:39.260225 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:39.260227 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:39.260248 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:39.261670 (+ 1422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:39.262047 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:39.262050 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:39.262051 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:39.262142 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:39.262416 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:39.262418 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:39.262419 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:39.262626 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:39.262728 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:39.262935 (+ 207us) server_negotiation.cc:299] Negotiation successful
0924 04:08:39.263025 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260924 04:08:39.263665 3451 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:37940
I20260924 04:08:39.264570 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 3513
I20260924 04:08:39.270465 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 3647
I20260924 04:08:39.275979 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 3781
I20260924 04:08:39.281160 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 3419
2026-09-24T04:08:39Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2318 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222898807720-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:08:39 dist-test-slave-00ld krb5kdc[3922](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:08:39 dist-test-slave-00ld krb5kdc[3922](info): set up 2 sockets
Sep 24 04:08:39 dist-test-slave-00ld krb5kdc[3922](info): commencing operation
krb5kdc: starting...
W20260924 04:08:40.336823 392 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222898807720-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222898807720-392-0/krb5kdc/test-user.keytab.
Sep 24 04:08:40 dist-test-slave-00ld krb5kdc[3922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222920, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:08:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:08:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:08:40.459791 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222898807720-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222898807720-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222898807720-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44825
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44825
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:08:40.567621 3935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:08:40.567891 3935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:08:40.567931 3935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:08:40.571777 3935 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:08:40.571830 3935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:08:40.571852 3935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:08:40.571868 3935 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:08:40.571883 3935 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:08:40.576902 3935 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222898807720-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222898807720-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44825
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3935
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222898807720-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:08:40.577877 3935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:08:40.578826 3935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:08:40.587215 3940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:08:40.587225 3941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:08:40.587236 3943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:08:40.587534 3935 server_base.cc:1034] running on GCE node
I20260924 04:08:40.588191 3935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:08:40.589325 3935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:08:40.590512 3935 hybrid_clock.cc:648] HybridClock initialized: now 1790222920590489 us; error 46 us; skew 500 ppm
Sep 24 04:08:40 dist-test-slave-00ld krb5kdc[3922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222920, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:08:40.593263 3935 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:08:40.594157 3935 webserver.cc:460] Webserver started at http://127.0.98.62:36075/ using document root <none> and password file <none>
I20260924 04:08:40.594697 3935 fs_manager.cc:359] Metadata directory not provided
I20260924 04:08:40.594799 3935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:08:40.595041 3935 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:08:40.597085 3935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222898807720-392-0/minicluster-data/master-0/data/instance:
uuid: "eb42c77e085443d39c3c66aa71b40409"
format_stamp: "Formatted at 2026-09-24 04:08:40 on dist-test-slave-00ld"
server_key: "803929f488d2cbe3b3a68775476c3be0"
server_key_iv: "e42e8ccc9a36980c01407f9f121f69c5"
server_key_version: "encryptionkey@0"
I20260924 04:08:40.597615 3935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222898807720-392-0/minicluster-data/master-0/wal/instance:
uuid: "eb42c77e085443d39c3c66aa71b40409"
format_stamp: "Formatted at 2026-09-24 04:08:40 on dist-test-slave-00ld"
server_key: "803929f488d2cbe3b3a68775476c3be0"
server_key_iv: "e42e8ccc9a36980c01407f9f121f69c5"
server_key_version: "encryptionkey@0"
I20260924 04:08:40.601173 3935 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260924 04:08:40.603461 3950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:40.604629 3935 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:08:40.604817 3935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222898807720-392-0/minicluster-data/master-0/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222898807720-392-0/minicluster-data/master-0/wal
uuid: "eb42c77e085443d39c3c66aa71b40409"
format_stamp: "Formatted at 2026-09-24 04:08:40 on dist-test-slave-00ld"
server_key: "803929f488d2cbe3b3a68775476c3be0"
server_key_iv: "e42e8ccc9a36980c01407f9f121f69c5"
server_key_version: "encryptionkey@0"
I20260924 04:08:40.604930 3935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222898807720-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222898807720-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222898807720-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:08:40.629357 3935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:08:40.632242 3935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:08:40.632452 3935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:08:40.639350 3935 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44825
I20260924 04:08:40.639379 4002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44825 every 8 connection(s)
I20260924 04:08:40.640275 3935 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:08:40.642974 4003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:08:40.645881 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 3935
I20260924 04:08:40.645977 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222898807720-392-0/minicluster-data/master-0/wal/instance
I20260924 04:08:40.646253 392 external_mini_cluster.cc:1428] Setting key aa1303dea2f8e1c9998cad5f6d4611ca
I20260924 04:08:40.648872 4003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb42c77e085443d39c3c66aa71b40409: Bootstrap starting.
I20260924 04:08:40.651466 4003 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb42c77e085443d39c3c66aa71b40409: Neither blocks nor log segments found. Creating new log.
I20260924 04:08:40.652298 4003 log.cc:824] T 00000000000000000000000000000000 P eb42c77e085443d39c3c66aa71b40409: Log is configured to *not* fsync() on all Append() calls
I20260924 04:08:40.654271 4003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb42c77e085443d39c3c66aa71b40409: No bootstrap required, opened a new log
Sep 24 04:08:40 dist-test-slave-00ld krb5kdc[3922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222920, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:08:40.657703 4003 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb42c77e085443d39c3c66aa71b40409 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb42c77e085443d39c3c66aa71b40409" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44825 } }
I20260924 04:08:40.657953 4003 raft_consensus.cc:374] T 00000000000000000000000000000000 P eb42c77e085443d39c3c66aa71b40409 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:08:40.658039 4003 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb42c77e085443d39c3c66aa71b40409 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb42c77e085443d39c3c66aa71b40409, State: Initialized, Role: FOLLOWER
I20260924 04:08:40.658465 4003 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb42c77e085443d39c3c66aa71b40409 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb42c77e085443d39c3c66aa71b40409" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44825 } }
I20260924 04:08:40.658581 4003 raft_consensus.cc:388] T 00000000000000000000000000000000 P eb42c77e085443d39c3c66aa71b40409 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:08:40.658665 4003 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb42c77e085443d39c3c66aa71b40409 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:08:40.658764 4003 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb42c77e085443d39c3c66aa71b40409 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:08:40.659629 4003 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb42c77e085443d39c3c66aa71b40409 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb42c77e085443d39c3c66aa71b40409" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44825 } }
I20260924 04:08:40.659976 4003 leader_election.cc:302] T 00000000000000000000000000000000 P eb42c77e085443d39c3c66aa71b40409 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb42c77e085443d39c3c66aa71b40409; no voters:
I20260924 04:08:40.660322 4003 leader_election.cc:288] T 00000000000000000000000000000000 P eb42c77e085443d39c3c66aa71b40409 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:08:40.660470 4008 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb42c77e085443d39c3c66aa71b40409 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:08:40.660882 4008 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb42c77e085443d39c3c66aa71b40409 [term 1 LEADER]: Becoming Leader. State: Replica: eb42c77e085443d39c3c66aa71b40409, State: Running, Role: LEADER
I20260924 04:08:40.661234 4008 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb42c77e085443d39c3c66aa71b40409 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb42c77e085443d39c3c66aa71b40409" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44825 } }
I20260924 04:08:40.661665 4003 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb42c77e085443d39c3c66aa71b40409 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:08:40.661875 4006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:40.647653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48540 (local address 127.0.98.62:44825)
0924 04:08:40.648071 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:40.648090 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:40.648134 (+ 44us) server_negotiation.cc:407] Connection header received
0924 04:08:40.648973 (+ 839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:40.648997 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:40.649410 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:40.649767 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:40.650480 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:40.651943 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:40.652912 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:40.653381 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:40.653553 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:40.656226 (+ 2673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:40.656231 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:40.656250 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:40.656275 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:40.658692 (+ 2417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:40.659131 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:40.659136 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:40.659142 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:40.659272 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:40.659551 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:40.659556 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:40.659558 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:40.659996 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:40.660195 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:40.660505 (+ 310us) server_negotiation.cc:299] Negotiation successful
0924 04:08:40.660837 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":107,"threads_started":1}
I20260924 04:08:40.662539 4008 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb42c77e085443d39c3c66aa71b40409 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb42c77e085443d39c3c66aa71b40409. Latest consensus state: current_term: 1 leader_uuid: "eb42c77e085443d39c3c66aa71b40409" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb42c77e085443d39c3c66aa71b40409" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44825 } } }
I20260924 04:08:40.662667 4008 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb42c77e085443d39c3c66aa71b40409 [sys.catalog]: This master's current role is: LEADER
I20260924 04:08:40.662921 4010 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb42c77e085443d39c3c66aa71b40409 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eb42c77e085443d39c3c66aa71b40409" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb42c77e085443d39c3c66aa71b40409" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44825 } } }
I20260924 04:08:40.663012 4010 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb42c77e085443d39c3c66aa71b40409 [sys.catalog]: This master's current role is: LEADER
W20260924 04:08:40.666239 4023 catalog_manager.cc:1558] T 00000000000000000000000000000000 P eb42c77e085443d39c3c66aa71b40409: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:08:40.666352 4023 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:08:40.666424 4017 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:08:40.667105 4017 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:08:40.672907 4017 catalog_manager.cc:1347] Generated new cluster ID: 7941e32145c14ff7ab951316cd5fe271
I20260924 04:08:40.672994 4017 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:08:40.685029 4017 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:08:40.686013 4017 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:08:40.693413 4017 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eb42c77e085443d39c3c66aa71b40409: Generated new TSK 0
I20260924 04:08:40.694160 4017 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222898807720-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222898807720-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:08:40.729455 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222898807720-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222898807720-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222898807720-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222898807720-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44825
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222898807720-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:08:40.843741 4029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:08:40.844036 4029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:08:40.844080 4029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:08:40.848259 4029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:08:40.848341 4029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:08:40.848402 4029 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:08:40.853849 4029 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222898807720-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222898807720-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222898807720-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222898807720-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4029
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222898807720-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:08:40.855034 4029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:08:40.856120 4029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:08:40.866102 4029 server_base.cc:1034] running on GCE node
W20260924 04:08:40.866068 4035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:08:40.866052 4034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:08:40.866062 4037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:08:40.866693 4029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:08:40.867347 4029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:08:40.868510 4029 hybrid_clock.cc:648] HybridClock initialized: now 1790222920868489 us; error 31 us; skew 500 ppm
Sep 24 04:08:40 dist-test-slave-00ld krb5kdc[3922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222920, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:08:40.871411 4029 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:08:40.872349 4029 webserver.cc:460] Webserver started at http://127.0.98.1:36409/ using document root <none> and password file <none>
I20260924 04:08:40.872979 4029 fs_manager.cc:359] Metadata directory not provided
I20260924 04:08:40.873070 4029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:08:40.873304 4029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:08:40.875325 4029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222898807720-392-0/minicluster-data/ts-0/data/instance:
uuid: "b927fa25bb044723a7873fd68a63b03a"
format_stamp: "Formatted at 2026-09-24 04:08:40 on dist-test-slave-00ld"
server_key: "8fffe8d75e526266f7dcc4b40e514b69"
server_key_iv: "cab0a506ac17461c72d88a1827555ce7"
server_key_version: "encryptionkey@0"
I20260924 04:08:40.875882 4029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222898807720-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b927fa25bb044723a7873fd68a63b03a"
format_stamp: "Formatted at 2026-09-24 04:08:40 on dist-test-slave-00ld"
server_key: "8fffe8d75e526266f7dcc4b40e514b69"
server_key_iv: "cab0a506ac17461c72d88a1827555ce7"
server_key_version: "encryptionkey@0"
I20260924 04:08:40.879693 4029 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260924 04:08:40.882345 4044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:40.883774 4029 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 04:08:40.883894 4029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222898807720-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222898807720-392-0/minicluster-data/ts-0/wal
uuid: "b927fa25bb044723a7873fd68a63b03a"
format_stamp: "Formatted at 2026-09-24 04:08:40 on dist-test-slave-00ld"
server_key: "8fffe8d75e526266f7dcc4b40e514b69"
server_key_iv: "cab0a506ac17461c72d88a1827555ce7"
server_key_version: "encryptionkey@0"
I20260924 04:08:40.884011 4029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222898807720-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222898807720-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222898807720-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:08:40.910430 4029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:08:40.913565 4029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:08:40.913717 4029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:08:40.914237 4029 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:08:40.915004 4029 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:08:40.915051 4029 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:40.915091 4029 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:08:40.915107 4029 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:40.923903 4029 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41947
I20260924 04:08:40.923920 4157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41947 every 8 connection(s)
I20260924 04:08:40.924921 4029 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222898807720-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:08:40.925768 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 4029
I20260924 04:08:40.925899 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222898807720-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:08:40.926229 392 external_mini_cluster.cc:1428] Setting key a5d5c2fd7478484cddf6ee9e247b6143
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Sep 24 04:08:40 dist-test-slave-00ld krb5kdc[3922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222920, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 04:08:40.941380 4006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:40.927052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52881 (local address 127.0.98.62:44825)
0924 04:08:40.927224 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:40.927228 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:40.928128 (+ 900us) server_negotiation.cc:407] Connection header received
0924 04:08:40.930121 (+ 1993us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:40.930126 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:40.930191 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:40.930299 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:40.931535 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:40.932507 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:40.933751 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:40.934153 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:40.934287 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:40.937017 (+ 2730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:40.937020 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:40.937022 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:40.937048 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:40.938617 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:40.939428 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:40.939432 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:40.939433 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:40.939518 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:40.939992 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:40.939995 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:40.939996 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:40.940196 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:40.940280 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:40.941035 (+ 755us) server_negotiation.cc:299] Negotiation successful
0924 04:08:40.941177 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 04:08:40.942332 4160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:40.927383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44825 (local address 127.0.98.1:52881)
0924 04:08:40.927950 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:40.928011 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:40.929878 (+ 1867us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:40.930470 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:40.930478 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:40.930862 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:08:40.931369 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:40.931383 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:40.932649 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:40.932652 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:40.933613 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:40.933622 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:40.934323 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:40.934325 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:40.934477 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:40.935132 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:08:40.935148 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:08:40.936817 (+ 1669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:08:40.938833 (+ 2016us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:40.938839 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:40.938898 (+ 59us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:40.939290 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:40.939619 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:40.939625 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:40.939630 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:40.939870 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:40.940302 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:08:40.940312 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:08:40.940750 (+ 438us) client_negotiation.cc:769] Sending connection context
0924 04:08:40.940971 (+ 221us) client_negotiation.cc:240] Negotiation successful
0924 04:08:40.941328 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":178,"threads_started":1}
I20260924 04:08:40.943823 4158 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44825
I20260924 04:08:40.944095 4158 heartbeater.cc:464] Registering TS with master...
I20260924 04:08:40.944633 4158 heartbeater.cc:511] Master 127.0.98.62:44825 requested a full tablet report, sending...
I20260924 04:08:40.946190 3967 ts_manager.cc:194] Registered new tserver with Master: b927fa25bb044723a7873fd68a63b03a (127.0.98.1:41947)
I20260924 04:08:40.947330 3967 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52881
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222898807720-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222898807720-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:08:40.956385 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222898807720-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222898807720-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222898807720-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222898807720-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44825
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222898807720-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:08:41.065917 4163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:08:41.066184 4163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:08:41.066268 4163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:08:41.070323 4163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:08:41.070382 4163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:08:41.070493 4163 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:08:41.076092 4163 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222898807720-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222898807720-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222898807720-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222898807720-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4163
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222898807720-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:08:41.077179 4163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:08:41.078218 4163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:08:41.087795 4168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:08:41.087963 4163 server_base.cc:1034] running on GCE node
W20260924 04:08:41.087976 4171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:08:41.088140 4169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:08:41.088609 4163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:08:41.089267 4163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:08:41.090427 4163 hybrid_clock.cc:648] HybridClock initialized: now 1790222921090408 us; error 31 us; skew 500 ppm
Sep 24 04:08:41 dist-test-slave-00ld krb5kdc[3922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222921, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:08:41.093050 4163 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:08:41.093992 4163 webserver.cc:460] Webserver started at http://127.0.98.2:43705/ using document root <none> and password file <none>
I20260924 04:08:41.094553 4163 fs_manager.cc:359] Metadata directory not provided
I20260924 04:08:41.094642 4163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:08:41.094846 4163 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:08:41.096877 4163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222898807720-392-0/minicluster-data/ts-1/data/instance:
uuid: "9fa801762de145a0b04db7e468419c36"
format_stamp: "Formatted at 2026-09-24 04:08:41 on dist-test-slave-00ld"
server_key: "90fc02c4359c4a642a490bb94fc4f81a"
server_key_iv: "a925b7b6c09ac941169f5b9fc21245bf"
server_key_version: "encryptionkey@0"
I20260924 04:08:41.097425 4163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222898807720-392-0/minicluster-data/ts-1/wal/instance:
uuid: "9fa801762de145a0b04db7e468419c36"
format_stamp: "Formatted at 2026-09-24 04:08:41 on dist-test-slave-00ld"
server_key: "90fc02c4359c4a642a490bb94fc4f81a"
server_key_iv: "a925b7b6c09ac941169f5b9fc21245bf"
server_key_version: "encryptionkey@0"
I20260924 04:08:41.101020 4163 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260924 04:08:41.103408 4178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:41.104635 4163 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:08:41.104796 4163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222898807720-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222898807720-392-0/minicluster-data/ts-1/wal
uuid: "9fa801762de145a0b04db7e468419c36"
format_stamp: "Formatted at 2026-09-24 04:08:41 on dist-test-slave-00ld"
server_key: "90fc02c4359c4a642a490bb94fc4f81a"
server_key_iv: "a925b7b6c09ac941169f5b9fc21245bf"
server_key_version: "encryptionkey@0"
I20260924 04:08:41.104908 4163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222898807720-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222898807720-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222898807720-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:08:41.119947 4163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:08:41.122962 4163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:08:41.123185 4163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:08:41.123782 4163 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:08:41.124590 4163 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:08:41.124666 4163 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:41.124748 4163 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:08:41.124796 4163 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:41.133775 4163 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39691
I20260924 04:08:41.133800 4291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39691 every 8 connection(s)
I20260924 04:08:41.134642 4163 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222898807720-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:08:41.142851 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 4163
I20260924 04:08:41.143008 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222898807720-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:08:41.143558 392 external_mini_cluster.cc:1428] Setting key bad628ee1fb6604e0063219365eed230
Sep 24 04:08:41 dist-test-slave-00ld krb5kdc[3922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222921, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:08:41.149533 4006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:41.136511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56699 (local address 127.0.98.62:44825)
0924 04:08:41.136640 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:41.136644 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:41.137470 (+ 826us) server_negotiation.cc:407] Connection header received
0924 04:08:41.138921 (+ 1451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:41.138925 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:41.138986 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:41.139098 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:41.140268 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:41.140851 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:41.141978 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:41.142206 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:41.142328 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:41.145427 (+ 3099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:41.145430 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:41.145432 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:41.145453 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:41.147077 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:41.147715 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:41.147718 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:41.147719 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:41.147829 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:41.148157 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:41.148163 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:41.148168 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:41.148437 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:41.148540 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:41.149162 (+ 622us) server_negotiation.cc:299] Negotiation successful
0924 04:08:41.149307 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 04:08:41.150298 4294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:41.136881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44825 (local address 127.0.98.2:56699)
0924 04:08:41.137314 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:41.137356 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:41.138632 (+ 1276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:41.139279 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:41.139285 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:41.139642 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:08:41.140119 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:41.140132 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:41.140974 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:41.140980 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:41.141854 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:41.141863 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:41.142360 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:41.142364 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:41.142560 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:41.143349 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:08:41.143420 (+ 71us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:08:41.145237 (+ 1817us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:08:41.147194 (+ 1957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:41.147202 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:41.147222 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:41.147597 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:41.147929 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:41.147932 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:41.147933 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:41.148064 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:41.148536 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:08:41.148541 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:08:41.148890 (+ 349us) client_negotiation.cc:769] Sending connection context
0924 04:08:41.149085 (+ 195us) client_negotiation.cc:240] Negotiation successful
0924 04:08:41.149390 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":105,"threads_started":1}
I20260924 04:08:41.151542 4292 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44825
I20260924 04:08:41.151826 4292 heartbeater.cc:464] Registering TS with master...
I20260924 04:08:41.152464 4292 heartbeater.cc:511] Master 127.0.98.62:44825 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 04:08:41.153735 3967 ts_manager.cc:194] Registered new tserver with Master: 9fa801762de145a0b04db7e468419c36 (127.0.98.2:39691)
I20260924 04:08:41.154342 3967 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:56699
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222898807720-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222898807720-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:08:41.171818 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222898807720-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222898807720-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222898807720-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222898807720-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44825
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222898807720-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:08:41.282462 4297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:08:41.282750 4297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:08:41.282793 4297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:08:41.287250 4297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:08:41.287333 4297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:08:41.287456 4297 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:08:41.293037 4297 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222898807720-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222898807720-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222898807720-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222898807720-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4297
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222898807720-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:08:41.294173 4297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:08:41.295233 4297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:08:41.304979 4305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:08:41.305120 4297 server_base.cc:1034] running on GCE node
W20260924 04:08:41.304971 4303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:08:41.304971 4302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:08:41.305685 4297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:08:41.306376 4297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:08:41.307559 4297 hybrid_clock.cc:648] HybridClock initialized: now 1790222921307539 us; error 33 us; skew 500 ppm
Sep 24 04:08:41 dist-test-slave-00ld krb5kdc[3922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222921, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:08:41.310544 4297 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:08:41.311480 4297 webserver.cc:460] Webserver started at http://127.0.98.3:36751/ using document root <none> and password file <none>
I20260924 04:08:41.312060 4297 fs_manager.cc:359] Metadata directory not provided
I20260924 04:08:41.312150 4297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:08:41.312355 4297 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:08:41.314483 4297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222898807720-392-0/minicluster-data/ts-2/data/instance:
uuid: "f13dcc05890b4d87a287e7e5fead5ff6"
format_stamp: "Formatted at 2026-09-24 04:08:41 on dist-test-slave-00ld"
server_key: "2650c336fce545a253c9095b6ff58884"
server_key_iv: "c42a08ae63e0f42a4a337879a3408ab2"
server_key_version: "encryptionkey@0"
I20260924 04:08:41.315059 4297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222898807720-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f13dcc05890b4d87a287e7e5fead5ff6"
format_stamp: "Formatted at 2026-09-24 04:08:41 on dist-test-slave-00ld"
server_key: "2650c336fce545a253c9095b6ff58884"
server_key_iv: "c42a08ae63e0f42a4a337879a3408ab2"
server_key_version: "encryptionkey@0"
I20260924 04:08:41.318735 4297 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 04:08:41.321173 4312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:41.322345 4297 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260924 04:08:41.322475 4297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222898807720-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222898807720-392-0/minicluster-data/ts-2/wal
uuid: "f13dcc05890b4d87a287e7e5fead5ff6"
format_stamp: "Formatted at 2026-09-24 04:08:41 on dist-test-slave-00ld"
server_key: "2650c336fce545a253c9095b6ff58884"
server_key_iv: "c42a08ae63e0f42a4a337879a3408ab2"
server_key_version: "encryptionkey@0"
I20260924 04:08:41.322582 4297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222898807720-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222898807720-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222898807720-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:08:41.340909 4297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:08:41.344566 4297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:08:41.344794 4297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:08:41.345546 4297 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:08:41.346586 4297 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:08:41.346647 4297 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:41.346712 4297 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:08:41.346745 4297 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:41.355675 4297 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33817
I20260924 04:08:41.355749 4425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33817 every 8 connection(s)
I20260924 04:08:41.356585 4297 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222898807720-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:08:41.357959 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 4297
I20260924 04:08:41.358084 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790222898807720-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:08:41.358384 392 external_mini_cluster.cc:1428] Setting key 0c7ae91cd6cf6f8879e3237145dfa2ae
Sep 24 04:08:41 dist-test-slave-00ld krb5kdc[3922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222921, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:08:41.370100 4006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:41.358473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34625 (local address 127.0.98.62:44825)
0924 04:08:41.358611 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:41.358616 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:41.359564 (+ 948us) server_negotiation.cc:407] Connection header received
0924 04:08:41.361080 (+ 1516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:41.361083 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:41.361135 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:41.361211 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:41.362424 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:41.363055 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:41.364188 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:41.364425 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:41.364536 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:41.366957 (+ 2421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:41.366960 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:41.366962 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:41.366988 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:41.368230 (+ 1242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:41.368766 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:41.368769 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:41.368770 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:41.368845 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:41.369162 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:41.369164 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:41.369165 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:41.369343 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:41.369416 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:41.369873 (+ 457us) server_negotiation.cc:299] Negotiation successful
0924 04:08:41.369973 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 04:08:41.370992 4428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:41.358916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44825 (local address 127.0.98.3:34625)
0924 04:08:41.359397 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:41.359472 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:41.360876 (+ 1404us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:41.361382 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:41.361389 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:41.361774 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:08:41.362286 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:41.362300 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:41.363181 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:41.363185 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:41.364051 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:41.364060 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:41.364575 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:41.364579 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:41.364811 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:41.365362 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:08:41.365378 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:08:41.366798 (+ 1420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:08:41.368354 (+ 1556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:41.368358 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:41.368371 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:41.368621 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:41.368928 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:41.368931 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:41.368932 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:41.369073 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:41.369428 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:08:41.369433 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:08:41.369666 (+ 233us) client_negotiation.cc:769] Sending connection context
0924 04:08:41.369817 (+ 151us) client_negotiation.cc:240] Negotiation successful
0924 04:08:41.370063 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":135,"threads_started":1}
I20260924 04:08:41.372117 4426 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44825
I20260924 04:08:41.372375 4426 heartbeater.cc:464] Registering TS with master...
I20260924 04:08:41.373015 4426 heartbeater.cc:511] Master 127.0.98.62:44825 requested a full tablet report, sending...
I20260924 04:08:41.374001 3967 ts_manager.cc:194] Registered new tserver with Master: f13dcc05890b4d87a287e7e5fead5ff6 (127.0.98.3:33817)
I20260924 04:08:41.374567 3967 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34625
I20260924 04:08:41.383927 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 04:08:41 dist-test-slave-00ld krb5kdc[3922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222921, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 04:08:41 dist-test-slave-00ld krb5kdc[3922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222921, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 04:08:41.409828 4433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:41.400368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32886 (local address 127.0.98.1:41947)
0924 04:08:41.400733 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:41.400758 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:41.400782 (+ 24us) server_negotiation.cc:407] Connection header received
0924 04:08:41.400837 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:41.400842 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:41.401039 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:41.401167 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:41.401794 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:41.402581 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:41.403430 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:41.403795 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:41.403877 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:41.406194 (+ 2317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:41.406202 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:41.406215 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:41.406241 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:41.407968 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:41.408419 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:41.408425 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:41.408426 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:41.408544 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:41.408829 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:41.408831 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:41.408832 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:41.409101 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:41.409386 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:41.409498 (+ 112us) server_negotiation.cc:299] Negotiation successful
0924 04:08:41.409671 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":120,"threads_started":1}
Sep 24 04:08:41 dist-test-slave-00ld krb5kdc[3922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222921, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:08:41.420357 4006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:41.412740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48552 (local address 127.0.98.62:44825)
0924 04:08:41.412909 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:41.412913 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:41.412929 (+ 16us) server_negotiation.cc:407] Connection header received
0924 04:08:41.413044 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:41.413046 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:41.413085 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:41.413144 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:41.413702 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:41.414198 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:41.414923 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:41.415119 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:41.415193 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:41.417443 (+ 2250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:41.417446 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:41.417448 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:41.417471 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:41.418901 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:41.419245 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:41.419247 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:41.419248 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:41.419325 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:41.419533 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:41.419535 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:41.419536 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:41.419737 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:41.419839 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:41.420031 (+ 192us) server_negotiation.cc:299] Negotiation successful
0924 04:08:41.420121 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260924 04:08:41.420830 3967 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48552
I20260924 04:08:41.421461 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 4029
I20260924 04:08:41.426733 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 4163
I20260924 04:08:41.432044 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 4297
I20260924 04:08:41.437103 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 3935
2026-09-24T04:08:41Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2156 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222898807720-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:08:41 dist-test-slave-00ld krb5kdc[4438](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:08:41 dist-test-slave-00ld krb5kdc[4438](info): set up 2 sockets
Sep 24 04:08:41 dist-test-slave-00ld krb5kdc[4438](info): commencing operation
krb5kdc: starting...
W20260924 04:08:42.480892 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222898807720-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222898807720-392-0/krb5kdc/test-user.keytab.
Sep 24 04:08:42 dist-test-slave-00ld krb5kdc[4438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222922, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:08:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:08:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:08:42.603410 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222898807720-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222898807720-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222898807720-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44227
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44227
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:08:42.713835 4451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:08:42.714139 4451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:08:42.714221 4451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:08:42.718355 4451 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:08:42.718441 4451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:08:42.718480 4451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:08:42.718539 4451 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:08:42.718587 4451 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:08:42.723752 4451 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222898807720-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222898807720-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44227
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4451
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222898807720-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:08:42.724822 4451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:08:42.725834 4451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:08:42.733858 4456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:08:42.733845 4457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:08:42.733894 4459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:08:42.734091 4451 server_base.cc:1034] running on GCE node
I20260924 04:08:42.734719 4451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:08:42.735791 4451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:08:42.736950 4451 hybrid_clock.cc:648] HybridClock initialized: now 1790222922736930 us; error 32 us; skew 500 ppm
Sep 24 04:08:42 dist-test-slave-00ld krb5kdc[4438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222922, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:08:42.746721 4451 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:08:42.747541 4451 fs_manager.cc:359] Metadata directory not provided
I20260924 04:08:42.747611 4451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:08:42.747780 4451 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:08:42.749946 4451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222898807720-392-0/minicluster-data/master-0/data/instance:
uuid: "bc1b4f2692e04bd2942a37656b0e45f8"
format_stamp: "Formatted at 2026-09-24 04:08:42 on dist-test-slave-00ld"
server_key: "aaaf8fb6f73ad9e555da6e51023c4410"
server_key_iv: "0bf55eb5c59df74f0afc5546f48658b1"
server_key_version: "encryptionkey@0"
I20260924 04:08:42.750460 4451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222898807720-392-0/minicluster-data/master-0/wal/instance:
uuid: "bc1b4f2692e04bd2942a37656b0e45f8"
format_stamp: "Formatted at 2026-09-24 04:08:42 on dist-test-slave-00ld"
server_key: "aaaf8fb6f73ad9e555da6e51023c4410"
server_key_iv: "0bf55eb5c59df74f0afc5546f48658b1"
server_key_version: "encryptionkey@0"
I20260924 04:08:42.754151 4451 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:08:42.756556 4465 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:42.757889 4451 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:08:42.757992 4451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222898807720-392-0/minicluster-data/master-0/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222898807720-392-0/minicluster-data/master-0/wal
uuid: "bc1b4f2692e04bd2942a37656b0e45f8"
format_stamp: "Formatted at 2026-09-24 04:08:42 on dist-test-slave-00ld"
server_key: "aaaf8fb6f73ad9e555da6e51023c4410"
server_key_iv: "0bf55eb5c59df74f0afc5546f48658b1"
server_key_version: "encryptionkey@0"
I20260924 04:08:42.758112 4451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222898807720-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222898807720-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222898807720-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:08:42.771371 4451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:08:42.774536 4451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:08:42.774776 4451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:08:42.782836 4451 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44227
I20260924 04:08:42.783110 4517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44227 every 8 connection(s)
I20260924 04:08:42.783546 4451 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:08:42.786316 4518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:08:42.789633 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 4451
I20260924 04:08:42.789748 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222898807720-392-0/minicluster-data/master-0/wal/instance
I20260924 04:08:42.790035 392 external_mini_cluster.cc:1428] Setting key 8085a59cdd10f3cf7ff0447b28166e3a
I20260924 04:08:42.792243 4518 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc1b4f2692e04bd2942a37656b0e45f8: Bootstrap starting.
I20260924 04:08:42.794932 4518 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc1b4f2692e04bd2942a37656b0e45f8: Neither blocks nor log segments found. Creating new log.
I20260924 04:08:42.795771 4518 log.cc:824] T 00000000000000000000000000000000 P bc1b4f2692e04bd2942a37656b0e45f8: Log is configured to *not* fsync() on all Append() calls
I20260924 04:08:42.797962 4518 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc1b4f2692e04bd2942a37656b0e45f8: No bootstrap required, opened a new log
Sep 24 04:08:42 dist-test-slave-00ld krb5kdc[4438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222922, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:08:42.802448 4518 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc1b4f2692e04bd2942a37656b0e45f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc1b4f2692e04bd2942a37656b0e45f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44227 } }
I20260924 04:08:42.802713 4518 raft_consensus.cc:374] T 00000000000000000000000000000000 P bc1b4f2692e04bd2942a37656b0e45f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:08:42.802799 4518 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc1b4f2692e04bd2942a37656b0e45f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc1b4f2692e04bd2942a37656b0e45f8, State: Initialized, Role: FOLLOWER
I20260924 04:08:42.803308 4518 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc1b4f2692e04bd2942a37656b0e45f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc1b4f2692e04bd2942a37656b0e45f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44227 } }
I20260924 04:08:42.803486 4518 raft_consensus.cc:388] T 00000000000000000000000000000000 P bc1b4f2692e04bd2942a37656b0e45f8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:08:42.803555 4518 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc1b4f2692e04bd2942a37656b0e45f8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:08:42.803651 4518 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc1b4f2692e04bd2942a37656b0e45f8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:08:42.804771 4518 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc1b4f2692e04bd2942a37656b0e45f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc1b4f2692e04bd2942a37656b0e45f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44227 } }
I20260924 04:08:42.805121 4518 leader_election.cc:302] T 00000000000000000000000000000000 P bc1b4f2692e04bd2942a37656b0e45f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bc1b4f2692e04bd2942a37656b0e45f8; no voters:
I20260924 04:08:42.805456 4518 leader_election.cc:288] T 00000000000000000000000000000000 P bc1b4f2692e04bd2942a37656b0e45f8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:08:42.805590 4523 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc1b4f2692e04bd2942a37656b0e45f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:08:42.805775 4521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:42.791349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43450 (local address 127.0.98.62:44227)
0924 04:08:42.791744 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:42.791764 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:42.791805 (+ 41us) server_negotiation.cc:407] Connection header received
0924 04:08:42.792614 (+ 809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:42.792638 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:42.793070 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:42.793447 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:42.794244 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:42.795721 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:42.796613 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:42.797292 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:42.797444 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:42.800243 (+ 2799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:42.800249 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:42.800266 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:42.800297 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:42.802464 (+ 2167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:42.803023 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:42.803031 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:42.803037 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:42.803171 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:42.803434 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:42.803438 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:42.803442 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:42.803969 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:42.804157 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:42.804419 (+ 262us) server_negotiation.cc:299] Negotiation successful
0924 04:08:42.804752 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":100,"threads_started":1}
I20260924 04:08:42.805806 4523 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc1b4f2692e04bd2942a37656b0e45f8 [term 1 LEADER]: Becoming Leader. State: Replica: bc1b4f2692e04bd2942a37656b0e45f8, State: Running, Role: LEADER
I20260924 04:08:42.806131 4523 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc1b4f2692e04bd2942a37656b0e45f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc1b4f2692e04bd2942a37656b0e45f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44227 } }
I20260924 04:08:42.806656 4518 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc1b4f2692e04bd2942a37656b0e45f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:08:42.807289 4523 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc1b4f2692e04bd2942a37656b0e45f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc1b4f2692e04bd2942a37656b0e45f8. Latest consensus state: current_term: 1 leader_uuid: "bc1b4f2692e04bd2942a37656b0e45f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc1b4f2692e04bd2942a37656b0e45f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44227 } } }
I20260924 04:08:42.807410 4523 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc1b4f2692e04bd2942a37656b0e45f8 [sys.catalog]: This master's current role is: LEADER
I20260924 04:08:42.807843 4525 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc1b4f2692e04bd2942a37656b0e45f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bc1b4f2692e04bd2942a37656b0e45f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc1b4f2692e04bd2942a37656b0e45f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44227 } } }
I20260924 04:08:42.807945 4525 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc1b4f2692e04bd2942a37656b0e45f8 [sys.catalog]: This master's current role is: LEADER
W20260924 04:08:42.811486 4538 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bc1b4f2692e04bd2942a37656b0e45f8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:08:42.811586 4538 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:08:42.811654 4530 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:08:42.812354 4530 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:08:42.822942 4530 catalog_manager.cc:1347] Generated new cluster ID: ce1dedfe427c43d0988e3b9a6faec96e
I20260924 04:08:42.823025 4530 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:08:42.845073 4530 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:08:42.846067 4530 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:08:42.853068 4530 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bc1b4f2692e04bd2942a37656b0e45f8: Generated new TSK 0
I20260924 04:08:42.853823 4530 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222898807720-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222898807720-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:08:42.889333 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222898807720-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222898807720-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222898807720-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222898807720-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44227
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222898807720-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:08:43.015205 4544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:08:43.015522 4544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:08:43.015600 4544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:08:43.019742 4544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:08:43.019840 4544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:08:43.019938 4544 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:08:43.025292 4544 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222898807720-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222898807720-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222898807720-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222898807720-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4544
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222898807720-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:08:43.026345 4544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:08:43.027388 4544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:08:43.036494 4550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:08:43.036571 4552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:08:43.036515 4549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:08:43.036906 4544 server_base.cc:1034] running on GCE node
I20260924 04:08:43.037321 4544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:08:43.037957 4544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:08:43.039148 4544 hybrid_clock.cc:648] HybridClock initialized: now 1790222923039133 us; error 52 us; skew 500 ppm
Sep 24 04:08:43 dist-test-slave-00ld krb5kdc[4438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222923, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:08:43.041900 4544 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:08:43.042588 4544 fs_manager.cc:359] Metadata directory not provided
I20260924 04:08:43.042685 4544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:08:43.042923 4544 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:08:43.044989 4544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222898807720-392-0/minicluster-data/ts-0/data/instance:
uuid: "23c780d7c74349aea076e77f2c41bcbf"
format_stamp: "Formatted at 2026-09-24 04:08:43 on dist-test-slave-00ld"
server_key: "6f4c1def0f68a180fb35fbd19cba1730"
server_key_iv: "724f55fd49c568531ce08e62ece4540b"
server_key_version: "encryptionkey@0"
I20260924 04:08:43.045578 4544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222898807720-392-0/minicluster-data/ts-0/wal/instance:
uuid: "23c780d7c74349aea076e77f2c41bcbf"
format_stamp: "Formatted at 2026-09-24 04:08:43 on dist-test-slave-00ld"
server_key: "6f4c1def0f68a180fb35fbd19cba1730"
server_key_iv: "724f55fd49c568531ce08e62ece4540b"
server_key_version: "encryptionkey@0"
I20260924 04:08:43.049077 4544 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260924 04:08:43.051302 4558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:43.052415 4544 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:08:43.052546 4544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222898807720-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222898807720-392-0/minicluster-data/ts-0/wal
uuid: "23c780d7c74349aea076e77f2c41bcbf"
format_stamp: "Formatted at 2026-09-24 04:08:43 on dist-test-slave-00ld"
server_key: "6f4c1def0f68a180fb35fbd19cba1730"
server_key_iv: "724f55fd49c568531ce08e62ece4540b"
server_key_version: "encryptionkey@0"
I20260924 04:08:43.052654 4544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222898807720-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222898807720-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222898807720-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:08:43.065725 4544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:08:43.068641 4544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:08:43.068918 4544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:08:43.069522 4544 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:08:43.070451 4544 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:08:43.070508 4544 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:43.070555 4544 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:08:43.070592 4544 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:43.079106 4544 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44079
I20260924 04:08:43.079123 4671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44079 every 8 connection(s)
I20260924 04:08:43.079703 4544 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222898807720-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:08:43.086264 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 4544
I20260924 04:08:43.086392 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222898807720-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:08:43.086694 392 external_mini_cluster.cc:1428] Setting key 456637c525428baad11fd1fbb6903d1a
Sep 24 04:08:43 dist-test-slave-00ld krb5kdc[4438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222923, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:08:43.093690 4521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:43.081420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38393 (local address 127.0.98.62:44227)
0924 04:08:43.081554 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:43.081558 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:43.082404 (+ 846us) server_negotiation.cc:407] Connection header received
0924 04:08:43.083669 (+ 1265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:43.083672 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:43.083719 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:43.083794 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:43.084926 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:43.085457 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:43.086837 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:43.087053 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:43.087169 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:43.089872 (+ 2703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:43.089875 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:43.089877 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:43.089899 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:43.091409 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:43.092006 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:43.092012 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:43.092016 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:43.092114 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:43.092451 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:43.092457 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:43.092462 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:43.092772 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:43.092886 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:43.093348 (+ 462us) server_negotiation.cc:299] Negotiation successful
0924 04:08:43.093490 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 04:08:43.094383 4674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:43.081804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44227 (local address 127.0.98.1:38393)
0924 04:08:43.082239 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:43.082292 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:43.083482 (+ 1190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:43.083953 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:43.083959 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:43.084314 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:08:43.084805 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:43.084818 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:43.085559 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:43.085562 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:43.086729 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:43.086739 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:43.087207 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:43.087223 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:43.087391 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:43.088004 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:08:43.088019 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:08:43.089696 (+ 1677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:08:43.091535 (+ 1839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:43.091541 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:43.091556 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:43.091889 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:43.092214 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:43.092217 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:43.092219 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:43.092350 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:43.092876 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:08:43.092881 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:08:43.093104 (+ 223us) client_negotiation.cc:769] Sending connection context
0924 04:08:43.093263 (+ 159us) client_negotiation.cc:240] Negotiation successful
0924 04:08:43.093505 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":115,"threads_started":1}
I20260924 04:08:43.095567 4672 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44227
I20260924 04:08:43.095824 4672 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 04:08:43.096267 4672 heartbeater.cc:511] Master 127.0.98.62:44227 requested a full tablet report, sending...
I20260924 04:08:43.097733 4482 ts_manager.cc:194] Registered new tserver with Master: 23c780d7c74349aea076e77f2c41bcbf (127.0.98.1:44079)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 04:08:43.098891 4482 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38393
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222898807720-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222898807720-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:08:43.115260 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222898807720-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222898807720-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222898807720-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222898807720-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44227
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222898807720-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:08:43.225476 4677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:08:43.225735 4677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:08:43.225780 4677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:08:43.229681 4677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:08:43.229737 4677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:08:43.229796 4677 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:08:43.234906 4677 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222898807720-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222898807720-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222898807720-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222898807720-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4677
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222898807720-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:08:43.235862 4677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:08:43.236819 4677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:08:43.245963 4683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:08:43.245977 4685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:08:43.246011 4682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:08:43.246272 4677 server_base.cc:1034] running on GCE node
I20260924 04:08:43.246906 4677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:08:43.247812 4677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:08:43.248962 4677 hybrid_clock.cc:648] HybridClock initialized: now 1790222923248947 us; error 25 us; skew 500 ppm
Sep 24 04:08:43 dist-test-slave-00ld krb5kdc[4438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222923, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:08:43.251832 4677 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:08:43.252616 4677 fs_manager.cc:359] Metadata directory not provided
I20260924 04:08:43.252772 4677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:08:43.252950 4677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:08:43.254961 4677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222898807720-392-0/minicluster-data/ts-1/data/instance:
uuid: "c30886d3597f4ea998cd1faec39c1d54"
format_stamp: "Formatted at 2026-09-24 04:08:43 on dist-test-slave-00ld"
server_key: "7c01e5afb933dbce6b2fae6d8661ab8c"
server_key_iv: "23e907bd98d44ba2253cff2839007918"
server_key_version: "encryptionkey@0"
I20260924 04:08:43.255537 4677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222898807720-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c30886d3597f4ea998cd1faec39c1d54"
format_stamp: "Formatted at 2026-09-24 04:08:43 on dist-test-slave-00ld"
server_key: "7c01e5afb933dbce6b2fae6d8661ab8c"
server_key_iv: "23e907bd98d44ba2253cff2839007918"
server_key_version: "encryptionkey@0"
I20260924 04:08:43.259330 4677 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 04:08:43.261981 4691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:43.263314 4677 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 04:08:43.263449 4677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222898807720-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222898807720-392-0/minicluster-data/ts-1/wal
uuid: "c30886d3597f4ea998cd1faec39c1d54"
format_stamp: "Formatted at 2026-09-24 04:08:43 on dist-test-slave-00ld"
server_key: "7c01e5afb933dbce6b2fae6d8661ab8c"
server_key_iv: "23e907bd98d44ba2253cff2839007918"
server_key_version: "encryptionkey@0"
I20260924 04:08:43.263568 4677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222898807720-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222898807720-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222898807720-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:08:43.284229 4677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:08:43.287099 4677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:08:43.287302 4677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:08:43.287886 4677 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:08:43.288719 4677 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:08:43.288795 4677 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:43.288870 4677 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:08:43.288923 4677 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:43.298079 4677 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45829
I20260924 04:08:43.298098 4804 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45829 every 8 connection(s)
I20260924 04:08:43.298792 4677 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222898807720-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:08:43.301180 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 4677
I20260924 04:08:43.301311 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222898807720-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:08:43.301586 392 external_mini_cluster.cc:1428] Setting key 562bcf859319f1e441058447ac4b81a6
Sep 24 04:08:43 dist-test-slave-00ld krb5kdc[4438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222923, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260924 04:08:43.316259 4521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:43.300512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46393 (local address 127.0.98.62:44227)
0924 04:08:43.300650 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:43.300653 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:43.301632 (+ 979us) server_negotiation.cc:407] Connection header received
0924 04:08:43.303475 (+ 1843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:43.303481 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:43.303541 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:43.303633 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:43.305518 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:43.306417 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:43.308363 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:43.309069 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:43.309323 (+ 254us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:43.312521 (+ 3198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:43.312524 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:43.312526 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:43.312551 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:43.313961 (+ 1410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:43.314554 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:43.314558 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:43.314560 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:43.314648 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:43.315050 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:43.315053 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:43.315054 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:43.315251 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:43.315339 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:43.316021 (+ 682us) server_negotiation.cc:299] Negotiation successful
0924 04:08:43.316126 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 04:08:43.317349 4807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:43.300996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44227 (local address 127.0.98.2:46393)
0924 04:08:43.301469 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:43.301520 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:43.303191 (+ 1671us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:43.303954 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:43.303965 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:43.304536 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:08:43.305312 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:43.305338 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:43.306591 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:43.306596 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:43.308178 (+ 1582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:43.308188 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:43.309331 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:43.309337 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:43.309574 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:43.310315 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:08:43.310331 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:08:43.312347 (+ 2016us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:08:43.314113 (+ 1766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:43.314119 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:43.314136 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:43.314446 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:43.314757 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:43.314763 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:43.314765 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:43.314961 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:43.315371 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:08:43.315381 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:08:43.315762 (+ 381us) client_negotiation.cc:769] Sending connection context
0924 04:08:43.315987 (+ 225us) client_negotiation.cc:240] Negotiation successful
0924 04:08:43.316369 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":132,"threads_started":1}
I20260924 04:08:43.318670 4805 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44227
I20260924 04:08:43.318936 4805 heartbeater.cc:464] Registering TS with master...
I20260924 04:08:43.319403 4805 heartbeater.cc:511] Master 127.0.98.62:44227 requested a full tablet report, sending...
I20260924 04:08:43.320453 4482 ts_manager.cc:194] Registered new tserver with Master: c30886d3597f4ea998cd1faec39c1d54 (127.0.98.2:45829)
I20260924 04:08:43.321089 4482 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:46393
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222898807720-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222898807720-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:08:43.331990 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222898807720-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222898807720-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222898807720-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222898807720-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44227
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222898807720-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:08:43.443567 4810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:08:43.443856 4810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:08:43.443933 4810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:08:43.447988 4810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:08:43.448092 4810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:08:43.448189 4810 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:08:43.453363 4810 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222898807720-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222898807720-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222898807720-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222898807720-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4810
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222898807720-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:08:43.454396 4810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:08:43.455391 4810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:08:43.464942 4818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:08:43.465008 4810 server_base.cc:1034] running on GCE node
W20260924 04:08:43.464942 4816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:08:43.464944 4815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:08:43.465651 4810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:08:43.466339 4810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:08:43.467525 4810 hybrid_clock.cc:648] HybridClock initialized: now 1790222923467511 us; error 26 us; skew 500 ppm
Sep 24 04:08:43 dist-test-slave-00ld krb5kdc[4438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222923, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:08:43.470489 4810 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:08:43.471249 4810 fs_manager.cc:359] Metadata directory not provided
I20260924 04:08:43.471343 4810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:08:43.471583 4810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:08:43.473747 4810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222898807720-392-0/minicluster-data/ts-2/data/instance:
uuid: "94ec5e239e2043fcbec172ab540aae30"
format_stamp: "Formatted at 2026-09-24 04:08:43 on dist-test-slave-00ld"
server_key: "a78e3f6cd5b043235ae4558e7c1fcded"
server_key_iv: "c6e6346a0397f3e26e0db18e9aafb869"
server_key_version: "encryptionkey@0"
I20260924 04:08:43.474318 4810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222898807720-392-0/minicluster-data/ts-2/wal/instance:
uuid: "94ec5e239e2043fcbec172ab540aae30"
format_stamp: "Formatted at 2026-09-24 04:08:43 on dist-test-slave-00ld"
server_key: "a78e3f6cd5b043235ae4558e7c1fcded"
server_key_iv: "c6e6346a0397f3e26e0db18e9aafb869"
server_key_version: "encryptionkey@0"
I20260924 04:08:43.478042 4810 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:08:43.480391 4824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:43.481662 4810 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:08:43.481827 4810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222898807720-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222898807720-392-0/minicluster-data/ts-2/wal
uuid: "94ec5e239e2043fcbec172ab540aae30"
format_stamp: "Formatted at 2026-09-24 04:08:43 on dist-test-slave-00ld"
server_key: "a78e3f6cd5b043235ae4558e7c1fcded"
server_key_iv: "c6e6346a0397f3e26e0db18e9aafb869"
server_key_version: "encryptionkey@0"
I20260924 04:08:43.481940 4810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222898807720-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222898807720-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222898807720-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:08:43.509883 4810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:08:43.513232 4810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:08:43.513454 4810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:08:43.514072 4810 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:08:43.514911 4810 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:08:43.514983 4810 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:43.515060 4810 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:08:43.515101 4810 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:43.523969 4810 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34375
I20260924 04:08:43.523991 4937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34375 every 8 connection(s)
I20260924 04:08:43.524590 4810 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222898807720-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:08:43.528510 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 4810
I20260924 04:08:43.528612 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222898807720-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:08:43.528915 392 external_mini_cluster.cc:1428] Setting key 8da41546ff9a690970ce7fa45635e7c7
Sep 24 04:08:43 dist-test-slave-00ld krb5kdc[4438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222923, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:08:43.538002 4521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:43.526409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59065 (local address 127.0.98.62:44227)
0924 04:08:43.526560 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:43.526565 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:43.527480 (+ 915us) server_negotiation.cc:407] Connection header received
0924 04:08:43.529021 (+ 1541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:43.529025 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:43.529081 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:43.529179 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:43.530353 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:43.530992 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:43.532162 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:43.532392 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:43.532506 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:43.534932 (+ 2426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:43.534935 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:43.534937 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:43.534961 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:43.536160 (+ 1199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:43.536601 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:43.536603 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:43.536604 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:43.536675 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:43.537031 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:43.537034 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:43.537035 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:43.537205 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:43.537281 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:43.537780 (+ 499us) server_negotiation.cc:299] Negotiation successful
0924 04:08:43.537884 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 04:08:43.538856 4940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:43.526851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44227 (local address 127.0.98.3:59065)
0924 04:08:43.527321 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:43.527365 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:43.528807 (+ 1442us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:43.529361 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:43.529368 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:43.529730 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:08:43.530213 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:43.530226 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:43.531116 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:43.531127 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:43.532029 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:43.532037 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:43.532542 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:43.532546 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:43.532784 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:43.533326 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:08:43.533342 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:08:43.534785 (+ 1443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:08:43.536263 (+ 1478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:43.536267 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:43.536279 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:43.536518 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:43.536822 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:43.536824 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:43.536825 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:43.536952 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:43.537290 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:08:43.537295 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:08:43.537535 (+ 240us) client_negotiation.cc:769] Sending connection context
0924 04:08:43.537693 (+ 158us) client_negotiation.cc:240] Negotiation successful
0924 04:08:43.537936 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":120,"threads_started":1}
I20260924 04:08:43.540068 4938 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44227
I20260924 04:08:43.540310 4938 heartbeater.cc:464] Registering TS with master...
I20260924 04:08:43.540843 4938 heartbeater.cc:511] Master 127.0.98.62:44227 requested a full tablet report, sending...
I20260924 04:08:43.542006 4482 ts_manager.cc:194] Registered new tserver with Master: 94ec5e239e2043fcbec172ab540aae30 (127.0.98.3:34375)
I20260924 04:08:43.542693 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:08:43.542721 4482 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59065
I20260924 04:08:43.551350 4521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:43.544754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43464 (local address 127.0.98.62:44227)
0924 04:08:43.544869 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:43.544874 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:43.545053 (+ 179us) server_negotiation.cc:407] Connection header received
0924 04:08:43.545134 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:43.545137 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:43.545174 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:43.545237 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:43.545858 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:43.546330 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:43.547115 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:43.547311 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:43.547382 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:43.548570 (+ 1188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:43.548573 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:43.548575 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:43.548596 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:43.549950 (+ 1354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:43.550327 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:43.550329 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:43.550330 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:43.550399 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:43.550614 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:43.550616 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:43.550617 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:43.550851 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:43.550987 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:43.551151 (+ 164us) server_negotiation.cc:299] Negotiation successful
0924 04:08:43.551238 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 04:08:43.554368 4482 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43464:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:08:43.557793 4482 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:08:43.572444 4951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:43.566207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50084 (local address 127.0.98.1:44079)
0924 04:08:43.567052 (+ 845us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:43.567059 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:43.567083 (+ 24us) server_negotiation.cc:407] Connection header received
0924 04:08:43.567169 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:43.567178 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:43.567391 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:43.567964 (+ 573us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:08:43.568124 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:43.569313 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:43.571133 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:43.571894 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:43.572023 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:43.572130 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:43.572224 (+ 94us) server_negotiation.cc:299] Negotiation successful
0924 04:08:43.572299 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":726,"thread_start_us":110,"threads_started":1}
I20260924 04:08:43.572444 4953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:43.566918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36600 (local address 127.0.98.2:45829)
0924 04:08:43.567453 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:43.567467 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:43.567489 (+ 22us) server_negotiation.cc:407] Connection header received
0924 04:08:43.567564 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:43.567569 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:43.567753 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:43.568312 (+ 559us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:08:43.568460 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:43.569537 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:43.570911 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:43.571691 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:43.571982 (+ 291us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:43.572125 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:43.572192 (+ 67us) server_negotiation.cc:299] Negotiation successful
0924 04:08:43.572281 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":102,"threads_started":1}
I20260924 04:08:43.572428 4949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:43.566161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45829 (local address 127.0.0.1:36600)
0924 04:08:43.566611 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:43.566625 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:43.566762 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:43.567991 (+ 1229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:43.567993 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:43.568012 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:08:43.568096 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:43.568101 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:43.569670 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:43.569673 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:43.570789 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:43.570797 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:43.571833 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:43.571836 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:43.571963 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:43.572009 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 04:08:43.572105 (+ 96us) client_negotiation.cc:240] Negotiation successful
0924 04:08:43.572238 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"mutex_wait_us":45,"spinlock_wait_cycles":6400,"thread_start_us":101,"threads_started":1}
I20260924 04:08:43.572428 4950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:43.566094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44079 (local address 127.0.0.1:50084)
0924 04:08:43.566512 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:43.566561 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:43.566761 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:43.567566 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:43.567573 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:43.567596 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:08:43.567801 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:43.567814 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:43.569442 (+ 1628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:43.569445 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:43.571002 (+ 1557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:43.571013 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:43.572028 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:43.572031 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:43.572121 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:43.572139 (+ 18us) client_negotiation.cc:769] Sending connection context
0924 04:08:43.572203 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 04:08:43.572252 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"mutex_wait_us":214,"spinlock_wait_cycles":7296,"thread_start_us":99,"threads_started":1}
I20260924 04:08:43.573756 4952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:43.566832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34375 (local address 127.0.0.1:45802)
0924 04:08:43.567072 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:43.567087 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:43.567176 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:43.568126 (+ 950us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:43.568129 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:43.568146 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:08:43.568248 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:43.568257 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:43.569312 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:43.569319 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:43.571002 (+ 1683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:43.571013 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:43.571801 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:43.571804 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:43.571902 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:43.573464 (+ 1562us) client_negotiation.cc:769] Sending connection context
0924 04:08:43.573536 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 04:08:43.573612 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"mutex_wait_us":131,"thread_start_us":119,"threads_started":1}
I20260924 04:08:43.574188 4954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:43.567281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45802 (local address 127.0.98.3:34375)
0924 04:08:43.567691 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:43.567702 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:43.567725 (+ 23us) server_negotiation.cc:407] Connection header received
0924 04:08:43.567798 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:43.567803 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:43.567987 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:43.568147 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:08:43.568371 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:43.569183 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:43.571144 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:43.571658 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:43.571790 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:43.571943 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:43.573926 (+ 1983us) server_negotiation.cc:299] Negotiation successful
0924 04:08:43.574027 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":132,"threads_started":1}
I20260924 04:08:43.575565 4739 tablet_service.cc:1467] Processing CreateTablet for tablet 08a14d6ec05a4e2cab1b5fd78533f117 (DEFAULT_TABLE table=test-table [id=a2910283b54e44cfbd70bc153b39a972]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:08:43.575745 4606 tablet_service.cc:1467] Processing CreateTablet for tablet 08a14d6ec05a4e2cab1b5fd78533f117 (DEFAULT_TABLE table=test-table [id=a2910283b54e44cfbd70bc153b39a972]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:08:43.576470 4739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08a14d6ec05a4e2cab1b5fd78533f117. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:08:43.576762 4606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08a14d6ec05a4e2cab1b5fd78533f117. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:08:43.576723 4872 tablet_service.cc:1467] Processing CreateTablet for tablet 08a14d6ec05a4e2cab1b5fd78533f117 (DEFAULT_TABLE table=test-table [id=a2910283b54e44cfbd70bc153b39a972]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:08:43.577661 4872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08a14d6ec05a4e2cab1b5fd78533f117. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:08:43.582499 4955 tablet_bootstrap.cc:492] T 08a14d6ec05a4e2cab1b5fd78533f117 P c30886d3597f4ea998cd1faec39c1d54: Bootstrap starting.
I20260924 04:08:43.584290 4956 tablet_bootstrap.cc:492] T 08a14d6ec05a4e2cab1b5fd78533f117 P 23c780d7c74349aea076e77f2c41bcbf: Bootstrap starting.
I20260924 04:08:43.584623 4955 tablet_bootstrap.cc:654] T 08a14d6ec05a4e2cab1b5fd78533f117 P c30886d3597f4ea998cd1faec39c1d54: Neither blocks nor log segments found. Creating new log.
I20260924 04:08:43.584847 4957 tablet_bootstrap.cc:492] T 08a14d6ec05a4e2cab1b5fd78533f117 P 94ec5e239e2043fcbec172ab540aae30: Bootstrap starting.
I20260924 04:08:43.586282 4955 log.cc:824] T 08a14d6ec05a4e2cab1b5fd78533f117 P c30886d3597f4ea998cd1faec39c1d54: Log is configured to *not* fsync() on all Append() calls
I20260924 04:08:43.586472 4956 tablet_bootstrap.cc:654] T 08a14d6ec05a4e2cab1b5fd78533f117 P 23c780d7c74349aea076e77f2c41bcbf: Neither blocks nor log segments found. Creating new log.
I20260924 04:08:43.586849 4957 tablet_bootstrap.cc:654] T 08a14d6ec05a4e2cab1b5fd78533f117 P 94ec5e239e2043fcbec172ab540aae30: Neither blocks nor log segments found. Creating new log.
I20260924 04:08:43.587853 4956 log.cc:824] T 08a14d6ec05a4e2cab1b5fd78533f117 P 23c780d7c74349aea076e77f2c41bcbf: Log is configured to *not* fsync() on all Append() calls
I20260924 04:08:43.587965 4957 log.cc:824] T 08a14d6ec05a4e2cab1b5fd78533f117 P 94ec5e239e2043fcbec172ab540aae30: Log is configured to *not* fsync() on all Append() calls
I20260924 04:08:43.588078 4955 tablet_bootstrap.cc:492] T 08a14d6ec05a4e2cab1b5fd78533f117 P c30886d3597f4ea998cd1faec39c1d54: No bootstrap required, opened a new log
I20260924 04:08:43.588292 4955 ts_tablet_manager.cc:1397] T 08a14d6ec05a4e2cab1b5fd78533f117 P c30886d3597f4ea998cd1faec39c1d54: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 04:08:43.589537 4956 tablet_bootstrap.cc:492] T 08a14d6ec05a4e2cab1b5fd78533f117 P 23c780d7c74349aea076e77f2c41bcbf: No bootstrap required, opened a new log
I20260924 04:08:43.589762 4956 ts_tablet_manager.cc:1397] T 08a14d6ec05a4e2cab1b5fd78533f117 P 23c780d7c74349aea076e77f2c41bcbf: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 04:08:43.589836 4957 tablet_bootstrap.cc:492] T 08a14d6ec05a4e2cab1b5fd78533f117 P 94ec5e239e2043fcbec172ab540aae30: No bootstrap required, opened a new log
I20260924 04:08:43.590010 4957 ts_tablet_manager.cc:1397] T 08a14d6ec05a4e2cab1b5fd78533f117 P 94ec5e239e2043fcbec172ab540aae30: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 04:08:43.592211 4955 raft_consensus.cc:348] T 08a14d6ec05a4e2cab1b5fd78533f117 P c30886d3597f4ea998cd1faec39c1d54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23c780d7c74349aea076e77f2c41bcbf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44079 } } peers { permanent_uuid: "c30886d3597f4ea998cd1faec39c1d54" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45829 } } peers { permanent_uuid: "94ec5e239e2043fcbec172ab540aae30" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34375 } }
I20260924 04:08:43.592499 4955 raft_consensus.cc:374] T 08a14d6ec05a4e2cab1b5fd78533f117 P c30886d3597f4ea998cd1faec39c1d54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:08:43.592556 4955 raft_consensus.cc:729] T 08a14d6ec05a4e2cab1b5fd78533f117 P c30886d3597f4ea998cd1faec39c1d54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c30886d3597f4ea998cd1faec39c1d54, State: Initialized, Role: FOLLOWER
I20260924 04:08:43.592411 4956 raft_consensus.cc:348] T 08a14d6ec05a4e2cab1b5fd78533f117 P 23c780d7c74349aea076e77f2c41bcbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23c780d7c74349aea076e77f2c41bcbf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44079 } } peers { permanent_uuid: "c30886d3597f4ea998cd1faec39c1d54" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45829 } } peers { permanent_uuid: "94ec5e239e2043fcbec172ab540aae30" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34375 } }
I20260924 04:08:43.592702 4956 raft_consensus.cc:374] T 08a14d6ec05a4e2cab1b5fd78533f117 P 23c780d7c74349aea076e77f2c41bcbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:08:43.592765 4956 raft_consensus.cc:729] T 08a14d6ec05a4e2cab1b5fd78533f117 P 23c780d7c74349aea076e77f2c41bcbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23c780d7c74349aea076e77f2c41bcbf, State: Initialized, Role: FOLLOWER
I20260924 04:08:43.593212 4956 consensus_queue.cc:260] T 08a14d6ec05a4e2cab1b5fd78533f117 P 23c780d7c74349aea076e77f2c41bcbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23c780d7c74349aea076e77f2c41bcbf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44079 } } peers { permanent_uuid: "c30886d3597f4ea998cd1faec39c1d54" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45829 } } peers { permanent_uuid: "94ec5e239e2043fcbec172ab540aae30" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34375 } }
I20260924 04:08:43.593070 4955 consensus_queue.cc:260] T 08a14d6ec05a4e2cab1b5fd78533f117 P c30886d3597f4ea998cd1faec39c1d54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23c780d7c74349aea076e77f2c41bcbf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44079 } } peers { permanent_uuid: "c30886d3597f4ea998cd1faec39c1d54" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45829 } } peers { permanent_uuid: "94ec5e239e2043fcbec172ab540aae30" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34375 } }
I20260924 04:08:43.593705 4957 raft_consensus.cc:348] T 08a14d6ec05a4e2cab1b5fd78533f117 P 94ec5e239e2043fcbec172ab540aae30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23c780d7c74349aea076e77f2c41bcbf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44079 } } peers { permanent_uuid: "c30886d3597f4ea998cd1faec39c1d54" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45829 } } peers { permanent_uuid: "94ec5e239e2043fcbec172ab540aae30" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34375 } }
I20260924 04:08:43.593952 4957 raft_consensus.cc:374] T 08a14d6ec05a4e2cab1b5fd78533f117 P 94ec5e239e2043fcbec172ab540aae30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:08:43.594004 4957 raft_consensus.cc:729] T 08a14d6ec05a4e2cab1b5fd78533f117 P 94ec5e239e2043fcbec172ab540aae30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94ec5e239e2043fcbec172ab540aae30, State: Initialized, Role: FOLLOWER
I20260924 04:08:43.594010 4672 heartbeater.cc:503] Master 127.0.98.62:44227 was elected leader, sending a full tablet report...
I20260924 04:08:43.594318 4956 ts_tablet_manager.cc:1428] T 08a14d6ec05a4e2cab1b5fd78533f117 P 23c780d7c74349aea076e77f2c41bcbf: Time spent starting tablet: real 0.004s user 0.002s sys 0.003s
I20260924 04:08:43.594388 4957 consensus_queue.cc:260] T 08a14d6ec05a4e2cab1b5fd78533f117 P 94ec5e239e2043fcbec172ab540aae30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23c780d7c74349aea076e77f2c41bcbf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44079 } } peers { permanent_uuid: "c30886d3597f4ea998cd1faec39c1d54" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45829 } } peers { permanent_uuid: "94ec5e239e2043fcbec172ab540aae30" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34375 } }
I20260924 04:08:43.594496 4955 ts_tablet_manager.cc:1428] T 08a14d6ec05a4e2cab1b5fd78533f117 P c30886d3597f4ea998cd1faec39c1d54: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 04:08:43.594818 4805 heartbeater.cc:503] Master 127.0.98.62:44227 was elected leader, sending a full tablet report...
I20260924 04:08:43.595052 4938 heartbeater.cc:503] Master 127.0.98.62:44227 was elected leader, sending a full tablet report...
I20260924 04:08:43.595424 4957 ts_tablet_manager.cc:1428] T 08a14d6ec05a4e2cab1b5fd78533f117 P 94ec5e239e2043fcbec172ab540aae30: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260924 04:08:43.671721 4963 raft_consensus.cc:482] T 08a14d6ec05a4e2cab1b5fd78533f117 P 94ec5e239e2043fcbec172ab540aae30 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:08:43.671942 4963 raft_consensus.cc:504] T 08a14d6ec05a4e2cab1b5fd78533f117 P 94ec5e239e2043fcbec172ab540aae30 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23c780d7c74349aea076e77f2c41bcbf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44079 } } peers { permanent_uuid: "c30886d3597f4ea998cd1faec39c1d54" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45829 } } peers { permanent_uuid: "94ec5e239e2043fcbec172ab540aae30" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34375 } }
I20260924 04:08:43.673228 4963 leader_election.cc:288] T 08a14d6ec05a4e2cab1b5fd78533f117 P 94ec5e239e2043fcbec172ab540aae30 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23c780d7c74349aea076e77f2c41bcbf (127.0.98.1:44079), c30886d3597f4ea998cd1faec39c1d54 (127.0.98.2:45829)
I20260924 04:08:43.676792 4940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:43.673414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44079 (local address 127.0.98.3:34083)
0924 04:08:43.673596 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:43.673613 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:43.673722 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:43.674006 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:43.674009 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:43.674025 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:08:43.674146 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:43.674151 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:43.674868 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:43.674873 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:43.675682 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:43.675691 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:43.676333 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:43.676336 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:43.676438 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:43.676464 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 04:08:43.676555 (+ 91us) client_negotiation.cc:240] Negotiation successful
0924 04:08:43.676636 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103}
I20260924 04:08:43.676792 4951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:43.673533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34083 (local address 127.0.98.1:44079)
0924 04:08:43.673667 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:43.673671 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:43.673688 (+ 17us) server_negotiation.cc:407] Connection header received
0924 04:08:43.673860 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:43.673863 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:43.673907 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:43.673995 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:08:43.674259 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:43.674754 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:43.675814 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:43.676211 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:43.676330 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:43.676392 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:43.676570 (+ 178us) server_negotiation.cc:299] Negotiation successful
0924 04:08:43.676635 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 04:08:43.676967 4953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:43.673620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38189 (local address 127.0.98.2:45829)
0924 04:08:43.673769 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:43.673774 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:43.673955 (+ 181us) server_negotiation.cc:407] Connection header received
0924 04:08:43.674113 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:43.674116 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:43.674159 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:43.674245 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:08:43.674503 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:43.674979 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:43.676011 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:43.676408 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:43.676522 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:43.676580 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:43.676789 (+ 209us) server_negotiation.cc:299] Negotiation successful
0924 04:08:43.676854 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 04:08:43.676980 4964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:43.673482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45829 (local address 127.0.98.3:38189)
0924 04:08:43.673860 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:43.673871 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:43.673990 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:43.674253 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:43.674256 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:43.674271 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:08:43.674391 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:43.674397 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:43.675095 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:43.675099 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:43.675883 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:43.675891 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:43.676525 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:43.676528 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:43.676632 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:43.676660 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 04:08:43.676772 (+ 112us) client_negotiation.cc:240] Negotiation successful
0924 04:08:43.676839 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":100,"threads_started":1}
I20260924 04:08:43.677536 4626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08a14d6ec05a4e2cab1b5fd78533f117" candidate_uuid: "94ec5e239e2043fcbec172ab540aae30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23c780d7c74349aea076e77f2c41bcbf" is_pre_election: true
I20260924 04:08:43.677600 4759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08a14d6ec05a4e2cab1b5fd78533f117" candidate_uuid: "94ec5e239e2043fcbec172ab540aae30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c30886d3597f4ea998cd1faec39c1d54" is_pre_election: true
I20260924 04:08:43.677842 4626 raft_consensus.cc:2454] T 08a14d6ec05a4e2cab1b5fd78533f117 P 23c780d7c74349aea076e77f2c41bcbf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94ec5e239e2043fcbec172ab540aae30 in term 0.
I20260924 04:08:43.677892 4759 raft_consensus.cc:2454] T 08a14d6ec05a4e2cab1b5fd78533f117 P c30886d3597f4ea998cd1faec39c1d54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94ec5e239e2043fcbec172ab540aae30 in term 0.
I20260924 04:08:43.678359 4828 leader_election.cc:302] T 08a14d6ec05a4e2cab1b5fd78533f117 P 94ec5e239e2043fcbec172ab540aae30 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23c780d7c74349aea076e77f2c41bcbf, 94ec5e239e2043fcbec172ab540aae30; no voters:
I20260924 04:08:43.678634 4963 raft_consensus.cc:2781] T 08a14d6ec05a4e2cab1b5fd78533f117 P 94ec5e239e2043fcbec172ab540aae30 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:08:43.678743 4963 raft_consensus.cc:482] T 08a14d6ec05a4e2cab1b5fd78533f117 P 94ec5e239e2043fcbec172ab540aae30 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:08:43.678805 4963 raft_consensus.cc:3037] T 08a14d6ec05a4e2cab1b5fd78533f117 P 94ec5e239e2043fcbec172ab540aae30 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:08:43.679826 4963 raft_consensus.cc:504] T 08a14d6ec05a4e2cab1b5fd78533f117 P 94ec5e239e2043fcbec172ab540aae30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23c780d7c74349aea076e77f2c41bcbf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44079 } } peers { permanent_uuid: "c30886d3597f4ea998cd1faec39c1d54" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45829 } } peers { permanent_uuid: "94ec5e239e2043fcbec172ab540aae30" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34375 } }
I20260924 04:08:43.680194 4963 leader_election.cc:288] T 08a14d6ec05a4e2cab1b5fd78533f117 P 94ec5e239e2043fcbec172ab540aae30 [CANDIDATE]: Term 1 election: Requested vote from peers 23c780d7c74349aea076e77f2c41bcbf (127.0.98.1:44079), c30886d3597f4ea998cd1faec39c1d54 (127.0.98.2:45829)
I20260924 04:08:43.680507 4626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08a14d6ec05a4e2cab1b5fd78533f117" candidate_uuid: "94ec5e239e2043fcbec172ab540aae30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23c780d7c74349aea076e77f2c41bcbf"
I20260924 04:08:43.680523 4759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08a14d6ec05a4e2cab1b5fd78533f117" candidate_uuid: "94ec5e239e2043fcbec172ab540aae30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c30886d3597f4ea998cd1faec39c1d54"
I20260924 04:08:43.680668 4759 raft_consensus.cc:3037] T 08a14d6ec05a4e2cab1b5fd78533f117 P c30886d3597f4ea998cd1faec39c1d54 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:08:43.680639 4626 raft_consensus.cc:3037] T 08a14d6ec05a4e2cab1b5fd78533f117 P 23c780d7c74349aea076e77f2c41bcbf [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:08:43.681681 4626 raft_consensus.cc:2454] T 08a14d6ec05a4e2cab1b5fd78533f117 P 23c780d7c74349aea076e77f2c41bcbf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94ec5e239e2043fcbec172ab540aae30 in term 1.
I20260924 04:08:43.681694 4759 raft_consensus.cc:2454] T 08a14d6ec05a4e2cab1b5fd78533f117 P c30886d3597f4ea998cd1faec39c1d54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94ec5e239e2043fcbec172ab540aae30 in term 1.
I20260924 04:08:43.681993 4825 leader_election.cc:302] T 08a14d6ec05a4e2cab1b5fd78533f117 P 94ec5e239e2043fcbec172ab540aae30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94ec5e239e2043fcbec172ab540aae30, c30886d3597f4ea998cd1faec39c1d54; no voters:
I20260924 04:08:43.682183 4963 raft_consensus.cc:2781] T 08a14d6ec05a4e2cab1b5fd78533f117 P 94ec5e239e2043fcbec172ab540aae30 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:08:43.682417 4963 raft_consensus.cc:686] T 08a14d6ec05a4e2cab1b5fd78533f117 P 94ec5e239e2043fcbec172ab540aae30 [term 1 LEADER]: Becoming Leader. State: Replica: 94ec5e239e2043fcbec172ab540aae30, State: Running, Role: LEADER
I20260924 04:08:43.682732 4963 consensus_queue.cc:237] T 08a14d6ec05a4e2cab1b5fd78533f117 P 94ec5e239e2043fcbec172ab540aae30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23c780d7c74349aea076e77f2c41bcbf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44079 } } peers { permanent_uuid: "c30886d3597f4ea998cd1faec39c1d54" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45829 } } peers { permanent_uuid: "94ec5e239e2043fcbec172ab540aae30" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34375 } }
I20260924 04:08:43.685956 4481 catalog_manager.cc:5495] T 08a14d6ec05a4e2cab1b5fd78533f117 P 94ec5e239e2043fcbec172ab540aae30 reported cstate change: term changed from 0 to 1, leader changed from <none> to 94ec5e239e2043fcbec172ab540aae30 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "94ec5e239e2043fcbec172ab540aae30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23c780d7c74349aea076e77f2c41bcbf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c30886d3597f4ea998cd1faec39c1d54" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94ec5e239e2043fcbec172ab540aae30" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34375 } health_report { overall_health: HEALTHY } } }
I20260924 04:08:43.698807 4954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:43.695171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45818 (local address 127.0.98.3:34375)
0924 04:08:43.695291 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:43.695298 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:43.695332 (+ 34us) server_negotiation.cc:407] Connection header received
0924 04:08:43.695520 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:43.695523 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:43.695582 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:43.695695 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:08:43.695937 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:43.696436 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:43.697370 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:43.697602 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:43.697708 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:43.698026 (+ 318us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:08:43.698411 (+ 385us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:08:43.698543 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:43.698601 (+ 58us) server_negotiation.cc:299] Negotiation successful
0924 04:08:43.698651 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 04:08:43.704461 4759 raft_consensus.cc:1261] T 08a14d6ec05a4e2cab1b5fd78533f117 P c30886d3597f4ea998cd1faec39c1d54 [term 1 FOLLOWER]: Refusing update from remote peer 94ec5e239e2043fcbec172ab540aae30: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:08:43.705130 4626 raft_consensus.cc:1261] T 08a14d6ec05a4e2cab1b5fd78533f117 P 23c780d7c74349aea076e77f2c41bcbf [term 1 FOLLOWER]: Refusing update from remote peer 94ec5e239e2043fcbec172ab540aae30: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:08:43.705243 4965 consensus_queue.cc:1035] T 08a14d6ec05a4e2cab1b5fd78533f117 P 94ec5e239e2043fcbec172ab540aae30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c30886d3597f4ea998cd1faec39c1d54" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:08:43.705760 4963 consensus_queue.cc:1035] T 08a14d6ec05a4e2cab1b5fd78533f117 P 94ec5e239e2043fcbec172ab540aae30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23c780d7c74349aea076e77f2c41bcbf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:08:43.712778 4968 mvcc.cc:205] Tried to move back new op lower bound from 7332753095486541824 to 7332753095407230976. Current Snapshot: MvccSnapshot[applied={T|T < 7332753095486541824}]
I20260924 04:08:43.714614 4970 mvcc.cc:205] Tried to move back new op lower bound from 7332753095486541824 to 7332753095407230976. Current Snapshot: MvccSnapshot[applied={T|T < 7332753095486541824}]
I20260924 04:08:43.716058 4969 mvcc.cc:205] Tried to move back new op lower bound from 7332753095486541824 to 7332753095407230976. Current Snapshot: MvccSnapshot[applied={T|T < 7332753095486541824 or (T in {7332753095486541824})}]
I20260924 04:08:43.721359 4482 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43464:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:08:43.721493 4482 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43464:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:08:43.723747 4482 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bc1b4f2692e04bd2942a37656b0e45f8: Sending DeleteTablet for 3 replicas of tablet 08a14d6ec05a4e2cab1b5fd78533f117
I20260924 04:08:43.724553 4606 tablet_service.cc:1514] Processing DeleteTablet for tablet 08a14d6ec05a4e2cab1b5fd78533f117 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:08:43 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50084
I20260924 04:08:43.724532 4872 tablet_service.cc:1514] Processing DeleteTablet for tablet 08a14d6ec05a4e2cab1b5fd78533f117 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:08:43 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45802
I20260924 04:08:43.725028 4978 tablet_replica.cc:330] T 08a14d6ec05a4e2cab1b5fd78533f117 P 94ec5e239e2043fcbec172ab540aae30: stopping tablet replica
I20260924 04:08:43.725035 4739 tablet_service.cc:1514] Processing DeleteTablet for tablet 08a14d6ec05a4e2cab1b5fd78533f117 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:08:43 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36600
I20260924 04:08:43.725349 4978 raft_consensus.cc:2229] T 08a14d6ec05a4e2cab1b5fd78533f117 P 94ec5e239e2043fcbec172ab540aae30 [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:08:43.725528 4980 tablet_replica.cc:330] T 08a14d6ec05a4e2cab1b5fd78533f117 P c30886d3597f4ea998cd1faec39c1d54: stopping tablet replica
I20260924 04:08:43.725731 4978 raft_consensus.cc:2258] T 08a14d6ec05a4e2cab1b5fd78533f117 P 94ec5e239e2043fcbec172ab540aae30 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:08:43.725890 4980 raft_consensus.cc:2229] T 08a14d6ec05a4e2cab1b5fd78533f117 P c30886d3597f4ea998cd1faec39c1d54 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:08:43.726085 4979 tablet_replica.cc:330] T 08a14d6ec05a4e2cab1b5fd78533f117 P 23c780d7c74349aea076e77f2c41bcbf: stopping tablet replica
I20260924 04:08:43.726243 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 4544
I20260924 04:08:43.726377 4979 raft_consensus.cc:2229] T 08a14d6ec05a4e2cab1b5fd78533f117 P 23c780d7c74349aea076e77f2c41bcbf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:08:43.728111 4980 raft_consensus.cc:2258] T 08a14d6ec05a4e2cab1b5fd78533f117 P c30886d3597f4ea998cd1faec39c1d54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:08:43.729710 4980 ts_tablet_manager.cc:1905] T 08a14d6ec05a4e2cab1b5fd78533f117 P c30886d3597f4ea998cd1faec39c1d54: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:08:43.730777 4978 ts_tablet_manager.cc:1905] T 08a14d6ec05a4e2cab1b5fd78533f117 P 94ec5e239e2043fcbec172ab540aae30: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:08:43.732396 4980 ts_tablet_manager.cc:1918] T 08a14d6ec05a4e2cab1b5fd78533f117 P c30886d3597f4ea998cd1faec39c1d54: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:08:43.732497 4980 log.cc:1196] T 08a14d6ec05a4e2cab1b5fd78533f117 P c30886d3597f4ea998cd1faec39c1d54: Deleting WAL directory at /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222898807720-392-0/minicluster-data/ts-1/wal/wals/08a14d6ec05a4e2cab1b5fd78533f117
I20260924 04:08:43.732874 4980 ts_tablet_manager.cc:1939] T 08a14d6ec05a4e2cab1b5fd78533f117 P c30886d3597f4ea998cd1faec39c1d54: Deleting consensus metadata
W20260924 04:08:43.733645 4469 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44079 (error 108)
I20260924 04:08:43.733989 4978 ts_tablet_manager.cc:1918] T 08a14d6ec05a4e2cab1b5fd78533f117 P 94ec5e239e2043fcbec172ab540aae30: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:08:43.734297 4466 catalog_manager.cc:4841] TS c30886d3597f4ea998cd1faec39c1d54 (127.0.98.2:45829): tablet 08a14d6ec05a4e2cab1b5fd78533f117 (table test-table [id=a2910283b54e44cfbd70bc153b39a972]) successfully deleted
I20260924 04:08:43.734448 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 4677
I20260924 04:08:43.735179 4978 log.cc:1196] T 08a14d6ec05a4e2cab1b5fd78533f117 P 94ec5e239e2043fcbec172ab540aae30: Deleting WAL directory at /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790222898807720-392-0/minicluster-data/ts-2/wal/wals/08a14d6ec05a4e2cab1b5fd78533f117
I20260924 04:08:43.735597 4978 ts_tablet_manager.cc:1939] T 08a14d6ec05a4e2cab1b5fd78533f117 P 94ec5e239e2043fcbec172ab540aae30: Deleting consensus metadata
I20260924 04:08:43.736135 4952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:43.735824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44079 (local address 127.0.0.1:50096)
0924 04:08:43.735968 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:43.736052 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260924 04:08:43.736382 4469 catalog_manager.cc:4584] TS 23c780d7c74349aea076e77f2c41bcbf (127.0.98.1:44079): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 08a14d6ec05a4e2cab1b5fd78533f117: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44079: connect: Connection refused (error 111)
I20260924 04:08:43.736714 4469 catalog_manager.cc:4841] TS 94ec5e239e2043fcbec172ab540aae30 (127.0.98.3:34375): tablet 08a14d6ec05a4e2cab1b5fd78533f117 (table test-table [id=a2910283b54e44cfbd70bc153b39a972]) successfully deleted
I20260924 04:08:43.741137 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 4810
I20260924 04:08:43.745903 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 4451
2026-09-24T04:08:43Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2308 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-24T04:08:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:08:43Z Disabled control of system clock
I20260924 04:08:43.775941 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222898807720-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222898807720-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222898807720-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38647
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:32809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38647
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 04:08:43.891858 4985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:08:43.892160 4985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:08:43.892198 4985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:08:43.896968 4985 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:08:43.897028 4985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:08:43.897044 4985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:08:43.897061 4985 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:08:43.897076 4985 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:08:43.902931 4985 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222898807720-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222898807720-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38647
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4985
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222898807720-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:08:43.904081 4985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:08:43.906011 4985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:08:43.915048 4993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:08:43.915064 4990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:08:43.915079 4991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:08:43.915387 4985 server_base.cc:1034] running on GCE node
I20260924 04:08:43.915956 4985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:08:43.917338 4985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:08:43.918534 4985 hybrid_clock.cc:648] HybridClock initialized: now 1790222923918496 us; error 54 us; skew 500 ppm
I20260924 04:08:43.920626 4985 webserver.cc:460] Webserver started at http://127.0.98.62:35203/ using document root <none> and password file <none>
I20260924 04:08:43.921293 4985 fs_manager.cc:359] Metadata directory not provided
I20260924 04:08:43.921381 4985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:08:43.921607 4985 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:08:43.924405 4985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222898807720-392-0/minicluster-data/master-0/data/instance:
uuid: "0814d00729fd4711b3bedb85216c831c"
format_stamp: "Formatted at 2026-09-24 04:08:43 on dist-test-slave-00ld"
server_key: "6b701b84fcd53a434738bcbc20a67023"
server_key_iv: "e27137914e6fcc6db0d9e0100804f04c"
server_key_version: "encryptionkey@0"
I20260924 04:08:43.925155 4985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222898807720-392-0/minicluster-data/master-0/wal/instance:
uuid: "0814d00729fd4711b3bedb85216c831c"
format_stamp: "Formatted at 2026-09-24 04:08:43 on dist-test-slave-00ld"
server_key: "6b701b84fcd53a434738bcbc20a67023"
server_key_iv: "e27137914e6fcc6db0d9e0100804f04c"
server_key_version: "encryptionkey@0"
I20260924 04:08:43.929736 4985 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 04:08:43.932358 4999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:43.933784 4985 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 04:08:43.933930 4985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222898807720-392-0/minicluster-data/master-0/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222898807720-392-0/minicluster-data/master-0/wal
uuid: "0814d00729fd4711b3bedb85216c831c"
format_stamp: "Formatted at 2026-09-24 04:08:43 on dist-test-slave-00ld"
server_key: "6b701b84fcd53a434738bcbc20a67023"
server_key_iv: "e27137914e6fcc6db0d9e0100804f04c"
server_key_version: "encryptionkey@0"
I20260924 04:08:43.934059 4985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222898807720-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222898807720-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222898807720-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:08:43.939288 4985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:08:43.939935 4985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:08:43.940146 4985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:08:43.948256 4985 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38647
I20260924 04:08:43.948261 5051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38647 every 8 connection(s)
I20260924 04:08:43.949357 4985 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:08:43.951740 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 4985
I20260924 04:08:43.951843 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222898807720-392-0/minicluster-data/master-0/wal/instance
I20260924 04:08:43.952203 392 external_mini_cluster.cc:1428] Setting key 415a31aed6ff10696d1296960a8c5a09
I20260924 04:08:43.952875 5052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:08:43.958587 5055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:43.953812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43732 (local address 127.0.98.62:38647)
0924 04:08:43.954456 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:43.954472 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:43.954509 (+ 37us) server_negotiation.cc:407] Connection header received
0924 04:08:43.955466 (+ 957us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:43.955485 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:43.955919 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:43.956237 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:43.957196 (+ 959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:43.957201 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:43.957219 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:08:43.957230 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:43.957289 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:08:43.957306 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:43.957418 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:43.957541 (+ 123us) server_negotiation.cc:299] Negotiation successful
0924 04:08:43.957850 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":137,"threads_started":1}
I20260924 04:08:43.959728 5052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0814d00729fd4711b3bedb85216c831c: Bootstrap starting.
I20260924 04:08:43.962145 5052 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0814d00729fd4711b3bedb85216c831c: Neither blocks nor log segments found. Creating new log.
I20260924 04:08:43.962838 5052 log.cc:824] T 00000000000000000000000000000000 P 0814d00729fd4711b3bedb85216c831c: Log is configured to *not* fsync() on all Append() calls
I20260924 04:08:43.964965 5052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0814d00729fd4711b3bedb85216c831c: No bootstrap required, opened a new log
I20260924 04:08:43.968807 5052 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0814d00729fd4711b3bedb85216c831c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0814d00729fd4711b3bedb85216c831c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38647 } }
I20260924 04:08:43.969060 5052 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0814d00729fd4711b3bedb85216c831c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:08:43.969105 5052 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0814d00729fd4711b3bedb85216c831c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0814d00729fd4711b3bedb85216c831c, State: Initialized, Role: FOLLOWER
I20260924 04:08:43.969482 5052 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0814d00729fd4711b3bedb85216c831c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0814d00729fd4711b3bedb85216c831c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38647 } }
I20260924 04:08:43.969596 5052 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0814d00729fd4711b3bedb85216c831c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:08:43.969645 5052 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0814d00729fd4711b3bedb85216c831c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:08:43.969722 5052 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0814d00729fd4711b3bedb85216c831c [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:08:43.987931 5052 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0814d00729fd4711b3bedb85216c831c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0814d00729fd4711b3bedb85216c831c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38647 } }
I20260924 04:08:43.988408 5052 leader_election.cc:302] T 00000000000000000000000000000000 P 0814d00729fd4711b3bedb85216c831c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0814d00729fd4711b3bedb85216c831c; no voters:
I20260924 04:08:43.988837 5052 leader_election.cc:288] T 00000000000000000000000000000000 P 0814d00729fd4711b3bedb85216c831c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:08:43.988981 5057 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0814d00729fd4711b3bedb85216c831c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:08:43.989259 5057 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0814d00729fd4711b3bedb85216c831c [term 1 LEADER]: Becoming Leader. State: Replica: 0814d00729fd4711b3bedb85216c831c, State: Running, Role: LEADER
I20260924 04:08:43.989660 5057 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0814d00729fd4711b3bedb85216c831c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0814d00729fd4711b3bedb85216c831c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38647 } }
I20260924 04:08:43.990254 5052 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0814d00729fd4711b3bedb85216c831c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:08:43.991387 5059 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0814d00729fd4711b3bedb85216c831c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0814d00729fd4711b3bedb85216c831c. Latest consensus state: current_term: 1 leader_uuid: "0814d00729fd4711b3bedb85216c831c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0814d00729fd4711b3bedb85216c831c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38647 } } }
I20260924 04:08:43.991529 5059 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0814d00729fd4711b3bedb85216c831c [sys.catalog]: This master's current role is: LEADER
I20260924 04:08:43.991868 5058 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0814d00729fd4711b3bedb85216c831c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0814d00729fd4711b3bedb85216c831c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0814d00729fd4711b3bedb85216c831c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38647 } } }
I20260924 04:08:43.991968 5058 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0814d00729fd4711b3bedb85216c831c [sys.catalog]: This master's current role is: LEADER
W20260924 04:08:43.995534 5072 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0814d00729fd4711b3bedb85216c831c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:08:43.995620 5072 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:08:43.995687 5066 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:08:43.996752 5066 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:08:44.001845 5066 catalog_manager.cc:1347] Generated new cluster ID: 32ca704abc0340489e6d1d802a7dfb1f
I20260924 04:08:44.001905 5066 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:08:44.012142 5066 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:08:44.012485 5066 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:08:44.025795 5066 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0814d00729fd4711b3bedb85216c831c: Generated new TSK 0
I20260924 04:08:44.026669 5066 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:08:44.040431 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222898807720-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222898807720-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222898807720-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222898807720-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38647
--builtin_ntp_servers=127.0.98.20:32809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 04:08:44.157964 5076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:08:44.158294 5076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:08:44.158370 5076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:08:44.162614 5076 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:08:44.162712 5076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:08:44.162811 5076 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:08:44.168004 5076 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222898807720-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222898807720-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222898807720-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5076
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222898807720-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:08:44.169651 5076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:08:44.171129 5076 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:08:44.181826 5084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:08:44.181815 5081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:08:44.181957 5076 server_base.cc:1034] running on GCE node
W20260924 04:08:44.181878 5082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:08:44.182641 5076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:08:44.183319 5076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:08:44.184513 5076 hybrid_clock.cc:648] HybridClock initialized: now 1790222924184487 us; error 40 us; skew 500 ppm
I20260924 04:08:44.186515 5076 webserver.cc:460] Webserver started at http://127.0.98.1:35635/ using document root <none> and password file <none>
I20260924 04:08:44.187120 5076 fs_manager.cc:359] Metadata directory not provided
I20260924 04:08:44.187223 5076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:08:44.187458 5076 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:08:44.189651 5076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222898807720-392-0/minicluster-data/ts-0/data/instance:
uuid: "24d8419acc004656b215237292bd6388"
format_stamp: "Formatted at 2026-09-24 04:08:44 on dist-test-slave-00ld"
server_key: "e4a6df5e79295e234ae71bf684e1edae"
server_key_iv: "5a159a931f38b15f2bc82b8f507585d6"
server_key_version: "encryptionkey@0"
I20260924 04:08:44.190299 5076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222898807720-392-0/minicluster-data/ts-0/wal/instance:
uuid: "24d8419acc004656b215237292bd6388"
format_stamp: "Formatted at 2026-09-24 04:08:44 on dist-test-slave-00ld"
server_key: "e4a6df5e79295e234ae71bf684e1edae"
server_key_iv: "5a159a931f38b15f2bc82b8f507585d6"
server_key_version: "encryptionkey@0"
I20260924 04:08:44.194458 5076 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260924 04:08:44.196913 5090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:44.198151 5076 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 04:08:44.198302 5076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222898807720-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222898807720-392-0/minicluster-data/ts-0/wal
uuid: "24d8419acc004656b215237292bd6388"
format_stamp: "Formatted at 2026-09-24 04:08:44 on dist-test-slave-00ld"
server_key: "e4a6df5e79295e234ae71bf684e1edae"
server_key_iv: "5a159a931f38b15f2bc82b8f507585d6"
server_key_version: "encryptionkey@0"
I20260924 04:08:44.198417 5076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222898807720-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222898807720-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222898807720-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:08:44.203189 5076 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:08:44.203804 5076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:08:44.203989 5076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:08:44.204545 5076 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:08:44.205507 5076 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:08:44.205580 5076 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:44.205651 5076 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:08:44.205708 5076 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:44.214764 5076 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41511
I20260924 04:08:44.214809 5203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41511 every 8 connection(s)
I20260924 04:08:44.215688 5076 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222898807720-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:08:44.216954 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 5076
I20260924 04:08:44.217103 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222898807720-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:08:44.217409 392 external_mini_cluster.cc:1428] Setting key ce8cf5745303740960cd31dcaecbc784
I20260924 04:08:44.219453 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222898807720-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222898807720-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222898807720-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222898807720-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38647
--builtin_ntp_servers=127.0.98.20:32809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260924 04:08:44.222647 5055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:44.217863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46381 (local address 127.0.98.62:38647)
0924 04:08:44.217993 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:44.217999 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:44.219274 (+ 1275us) server_negotiation.cc:407] Connection header received
0924 04:08:44.220752 (+ 1478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:44.220758 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:44.220808 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:44.220899 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:44.222226 (+ 1327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:44.222229 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:44.222231 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:08:44.222242 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:44.222263 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:08:44.222271 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:44.222345 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:44.222511 (+ 166us) server_negotiation.cc:299] Negotiation successful
0924 04:08:44.222548 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 04:08:44.223819 5206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:44.218361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38647 (local address 127.0.98.1:46381)
0924 04:08:44.219131 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:44.219172 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:44.220503 (+ 1331us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:44.221111 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:44.221123 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:44.221279 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:08:44.221901 (+ 622us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:08:44.221905 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:08:44.221961 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:08:44.222069 (+ 108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:08:44.222357 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:08:44.222368 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:08:44.222376 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 04:08:44.222548 (+ 172us) client_negotiation.cc:240] Negotiation successful
0924 04:08:44.222786 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":158,"threads_started":1}
I20260924 04:08:44.225129 5204 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38647
I20260924 04:08:44.225378 5204 heartbeater.cc:464] Registering TS with master...
I20260924 04:08:44.225785 5204 heartbeater.cc:511] Master 127.0.98.62:38647 requested a full tablet report, sending...
I20260924 04:08:44.227411 5016 ts_manager.cc:194] Registered new tserver with Master: 24d8419acc004656b215237292bd6388 (127.0.98.1:41511)
W20260924 04:08:44.335968 5207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:08:44.336244 5207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:08:44.336285 5207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:08:44.340536 5207 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:08:44.340601 5207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:08:44.340659 5207 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:08:44.345510 5207 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222898807720-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222898807720-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222898807720-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5207
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222898807720-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:08:44.346593 5207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:08:44.347978 5207 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:08:44.358249 5215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:08:44.358256 5213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:08:44.358263 5212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:08:44.358780 5207 server_base.cc:1034] running on GCE node
I20260924 04:08:44.359427 5207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:08:44.360188 5207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:08:44.361379 5207 hybrid_clock.cc:648] HybridClock initialized: now 1790222924361341 us; error 63 us; skew 500 ppm
I20260924 04:08:44.363523 5207 webserver.cc:460] Webserver started at http://127.0.98.2:41255/ using document root <none> and password file <none>
I20260924 04:08:44.364166 5207 fs_manager.cc:359] Metadata directory not provided
I20260924 04:08:44.364265 5207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:08:44.364511 5207 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:08:44.366912 5207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222898807720-392-0/minicluster-data/ts-1/data/instance:
uuid: "8d79a03d13d9402a80d48b7d81a5aed6"
format_stamp: "Formatted at 2026-09-24 04:08:44 on dist-test-slave-00ld"
server_key: "6b37ea90a7b9192b2e377e7f1b61118b"
server_key_iv: "c4e7cd7cc575f6e97caebb78f6de71e2"
server_key_version: "encryptionkey@0"
I20260924 04:08:44.367575 5207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222898807720-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8d79a03d13d9402a80d48b7d81a5aed6"
format_stamp: "Formatted at 2026-09-24 04:08:44 on dist-test-slave-00ld"
server_key: "6b37ea90a7b9192b2e377e7f1b61118b"
server_key_iv: "c4e7cd7cc575f6e97caebb78f6de71e2"
server_key_version: "encryptionkey@0"
I20260924 04:08:44.371767 5207 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 04:08:44.374514 5221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:44.375857 5207 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260924 04:08:44.376010 5207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222898807720-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222898807720-392-0/minicluster-data/ts-1/wal
uuid: "8d79a03d13d9402a80d48b7d81a5aed6"
format_stamp: "Formatted at 2026-09-24 04:08:44 on dist-test-slave-00ld"
server_key: "6b37ea90a7b9192b2e377e7f1b61118b"
server_key_iv: "c4e7cd7cc575f6e97caebb78f6de71e2"
server_key_version: "encryptionkey@0"
I20260924 04:08:44.376137 5207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222898807720-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222898807720-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222898807720-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:08:44.381145 5207 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:08:44.381791 5207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:08:44.381978 5207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:08:44.382534 5207 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:08:44.383443 5207 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:08:44.383515 5207 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:44.383584 5207 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:08:44.383634 5207 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:44.393141 5207 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33629
I20260924 04:08:44.393217 5334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33629 every 8 connection(s)
I20260924 04:08:44.394098 5207 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222898807720-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:08:44.396435 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 5207
I20260924 04:08:44.396603 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222898807720-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:08:44.397034 392 external_mini_cluster.cc:1428] Setting key 411dc0ba8d933301041d5455314b3ba1
I20260924 04:08:44.399477 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222898807720-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222898807720-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222898807720-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222898807720-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38647
--builtin_ntp_servers=127.0.98.20:32809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260924 04:08:44.401713 5055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:44.396068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58009 (local address 127.0.98.62:38647)
0924 04:08:44.396249 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:44.396254 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:44.397633 (+ 1379us) server_negotiation.cc:407] Connection header received
0924 04:08:44.399491 (+ 1858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:44.399495 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:44.399530 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:44.399625 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:44.401095 (+ 1470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:44.401101 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:44.401106 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:08:44.401122 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:44.401149 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:08:44.401161 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:44.401256 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:44.401529 (+ 273us) server_negotiation.cc:299] Negotiation successful
0924 04:08:44.401583 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 04:08:44.403059 5337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:44.396485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38647 (local address 127.0.98.2:58009)
0924 04:08:44.397436 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:44.397513 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:44.399242 (+ 1729us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:44.399923 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:44.399936 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:44.400083 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:08:44.400718 (+ 635us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:08:44.400723 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:08:44.400788 (+ 65us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:08:44.400890 (+ 102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:08:44.401253 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:08:44.401284 (+ 31us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:08:44.401293 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 04:08:44.401935 (+ 642us) client_negotiation.cc:240] Negotiation successful
0924 04:08:44.402164 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":199,"threads_started":1}
I20260924 04:08:44.404347 5335 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38647
I20260924 04:08:44.404618 5335 heartbeater.cc:464] Registering TS with master...
I20260924 04:08:44.405055 5335 heartbeater.cc:511] Master 127.0.98.62:38647 requested a full tablet report, sending...
I20260924 04:08:44.406172 5016 ts_manager.cc:194] Registered new tserver with Master: 8d79a03d13d9402a80d48b7d81a5aed6 (127.0.98.2:33629)
W20260924 04:08:44.513576 5338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:08:44.513877 5338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:08:44.513960 5338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:08:44.518046 5338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:08:44.518133 5338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:08:44.518229 5338 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:08:44.523187 5338 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222898807720-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222898807720-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222898807720-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5338
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222898807720-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:08:44.524344 5338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:08:44.525918 5338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:08:44.535538 5346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:08:44.535538 5343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:08:44.535538 5344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:08:44.535956 5338 server_base.cc:1034] running on GCE node
I20260924 04:08:44.536394 5338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:08:44.537190 5338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:08:44.538374 5338 hybrid_clock.cc:648] HybridClock initialized: now 1790222924538350 us; error 37 us; skew 500 ppm
I20260924 04:08:44.540279 5338 webserver.cc:460] Webserver started at http://127.0.98.3:45179/ using document root <none> and password file <none>
I20260924 04:08:44.540939 5338 fs_manager.cc:359] Metadata directory not provided
I20260924 04:08:44.541030 5338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:08:44.541268 5338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:08:44.543322 5338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222898807720-392-0/minicluster-data/ts-2/data/instance:
uuid: "e4fa37b9466a44f9a82d7f3a1dc5cd93"
format_stamp: "Formatted at 2026-09-24 04:08:44 on dist-test-slave-00ld"
server_key: "7a84d5ac6423819129c59f34a2435943"
server_key_iv: "807530ccb09eaf8b4d9fcd189231cd07"
server_key_version: "encryptionkey@0"
I20260924 04:08:44.543881 5338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222898807720-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e4fa37b9466a44f9a82d7f3a1dc5cd93"
format_stamp: "Formatted at 2026-09-24 04:08:44 on dist-test-slave-00ld"
server_key: "7a84d5ac6423819129c59f34a2435943"
server_key_iv: "807530ccb09eaf8b4d9fcd189231cd07"
server_key_version: "encryptionkey@0"
I20260924 04:08:44.547577 5338 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260924 04:08:44.550096 5352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:44.551406 5338 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:08:44.551543 5338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222898807720-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222898807720-392-0/minicluster-data/ts-2/wal
uuid: "e4fa37b9466a44f9a82d7f3a1dc5cd93"
format_stamp: "Formatted at 2026-09-24 04:08:44 on dist-test-slave-00ld"
server_key: "7a84d5ac6423819129c59f34a2435943"
server_key_iv: "807530ccb09eaf8b4d9fcd189231cd07"
server_key_version: "encryptionkey@0"
I20260924 04:08:44.551663 5338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222898807720-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222898807720-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222898807720-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:08:44.556627 5338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:08:44.557309 5338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:08:44.557507 5338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:08:44.558069 5338 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:08:44.558990 5338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:08:44.559060 5338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:44.559131 5338 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:08:44.559187 5338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:44.568279 5338 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:32781
I20260924 04:08:44.568338 5465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:32781 every 8 connection(s)
I20260924 04:08:44.569339 5338 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222898807720-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:08:44.575479 5055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:44.571191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45003 (local address 127.0.98.62:38647)
0924 04:08:44.571327 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:44.571330 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:44.572461 (+ 1131us) server_negotiation.cc:407] Connection header received
0924 04:08:44.573823 (+ 1362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:44.573827 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:44.573875 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:44.573961 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:44.575001 (+ 1040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:44.575004 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:44.575006 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:08:44.575018 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:44.575046 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:08:44.575054 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:44.575169 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:44.575299 (+ 130us) server_negotiation.cc:299] Negotiation successful
0924 04:08:44.575343 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 04:08:44.576344 5468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:44.571588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38647 (local address 127.0.98.3:45003)
0924 04:08:44.572306 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:44.572347 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:44.573612 (+ 1265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:44.574144 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:44.574150 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:44.574236 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:08:44.574749 (+ 513us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:08:44.574751 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:08:44.574792 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:08:44.574856 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:08:44.575145 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:08:44.575153 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:08:44.575159 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 04:08:44.575340 (+ 181us) client_negotiation.cc:240] Negotiation successful
0924 04:08:44.575587 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":133,"threads_started":1}
I20260924 04:08:44.576876 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 5338
I20260924 04:08:44.576996 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222898807720-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:08:44.577275 392 external_mini_cluster.cc:1428] Setting key 50aeff864e09abbb03efb51e88697369
I20260924 04:08:44.577605 5466 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38647
I20260924 04:08:44.577867 5466 heartbeater.cc:464] Registering TS with master...
I20260924 04:08:44.578346 5466 heartbeater.cc:511] Master 127.0.98.62:38647 requested a full tablet report, sending...
I20260924 04:08:44.579371 5016 ts_manager.cc:194] Registered new tserver with Master: e4fa37b9466a44f9a82d7f3a1dc5cd93 (127.0.98.3:32781)
I20260924 04:08:44.591202 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:08:44.594911 5055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:44.593255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43744 (local address 127.0.98.62:38647)
0924 04:08:44.593393 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:44.593398 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:44.593549 (+ 151us) server_negotiation.cc:407] Connection header received
0924 04:08:44.593653 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:44.593655 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:44.593687 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:44.593744 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:44.594476 (+ 732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:44.594479 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:44.594481 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:08:44.594492 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:44.594511 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:08:44.594519 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:44.594638 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:44.594695 (+ 57us) server_negotiation.cc:299] Negotiation successful
0924 04:08:44.594728 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 04:08:44.597571 5016 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43744:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:08:44.600535 5016 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:08:44.612095 5477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:44.609116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33629 (local address 127.0.0.1:40972)
0924 04:08:44.609910 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:44.609964 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:44.610067 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:44.610640 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:44.610645 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:44.610721 (+ 76us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:08:44.611299 (+ 578us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:08:44.611303 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:08:44.611326 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:08:44.611393 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:08:44.611719 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:08:44.611727 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:08:44.611733 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 04:08:44.611879 (+ 146us) client_negotiation.cc:240] Negotiation successful
0924 04:08:44.611966 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":705,"thread_start_us":104,"threads_started":1}
I20260924 04:08:44.612185 5480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:44.609439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40972 (local address 127.0.98.2:33629)
0924 04:08:44.609881 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:44.609892 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:44.610172 (+ 280us) server_negotiation.cc:407] Connection header received
0924 04:08:44.610235 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:44.610241 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:44.610459 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:44.610654 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:44.611523 (+ 869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:44.611530 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:44.611542 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:08:44.611554 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:44.611599 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:08:44.611622 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:44.611735 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:44.611907 (+ 172us) server_negotiation.cc:299] Negotiation successful
0924 04:08:44.611993 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":104,"threads_started":1}
I20260924 04:08:44.613124 5479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:44.609101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53188 (local address 127.0.98.3:32781)
0924 04:08:44.609688 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:44.609701 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:44.610069 (+ 368us) server_negotiation.cc:407] Connection header received
0924 04:08:44.610285 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:44.610292 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:44.610501 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:44.610632 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:44.612428 (+ 1796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:44.612434 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:44.612449 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:08:44.612462 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:44.612507 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:08:44.612543 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:44.612634 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:44.612846 (+ 212us) server_negotiation.cc:299] Negotiation successful
0924 04:08:44.612924 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":136,"threads_started":1}
I20260924 04:08:44.613272 5481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:44.609669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41511 (local address 127.0.0.1:35506)
0924 04:08:44.610465 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:44.610478 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:44.610559 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:44.611301 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:44.611307 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:44.611327 (+ 20us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:08:44.612498 (+ 1171us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:08:44.612500 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:08:44.612509 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:08:44.612531 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:08:44.612948 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:08:44.612951 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:08:44.612954 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 04:08:44.613022 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 04:08:44.613139 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":740,"thread_start_us":85,"threads_started":1}
I20260924 04:08:44.613492 5478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:44.608935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32781 (local address 127.0.0.1:53188)
0924 04:08:44.609911 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:44.609966 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:44.610147 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:44.611374 (+ 1227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:44.611378 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:44.611394 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:08:44.612243 (+ 849us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:08:44.612244 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:08:44.612256 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:08:44.612284 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:08:44.612722 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:08:44.612725 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:08:44.612729 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 04:08:44.613327 (+ 598us) client_negotiation.cc:240] Negotiation successful
0924 04:08:44.613365 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":799,"thread_start_us":87,"threads_started":1}
I20260924 04:08:44.613855 5482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:44.610058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35506 (local address 127.0.98.1:41511)
0924 04:08:44.610484 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:44.610495 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:44.610558 (+ 63us) server_negotiation.cc:407] Connection header received
0924 04:08:44.610685 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:44.610691 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:44.610845 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:44.611577 (+ 732us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:44.612701 (+ 1124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:44.612710 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:44.612722 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:08:44.612733 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:44.612775 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:08:44.612808 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:44.612910 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:44.613512 (+ 602us) server_negotiation.cc:299] Negotiation successful
0924 04:08:44.613695 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":146,"threads_started":1}
I20260924 04:08:44.615584 5269 tablet_service.cc:1467] Processing CreateTablet for tablet 98a09c8777844b45b1a1c78282920927 (DEFAULT_TABLE table=test-table [id=3e9cdb497482420e8a6ed7399b6278bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:08:44.616088 5138 tablet_service.cc:1467] Processing CreateTablet for tablet 98a09c8777844b45b1a1c78282920927 (DEFAULT_TABLE table=test-table [id=3e9cdb497482420e8a6ed7399b6278bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:08:44.616115 5400 tablet_service.cc:1467] Processing CreateTablet for tablet 98a09c8777844b45b1a1c78282920927 (DEFAULT_TABLE table=test-table [id=3e9cdb497482420e8a6ed7399b6278bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:08:44.616736 5269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98a09c8777844b45b1a1c78282920927. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:08:44.617121 5138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98a09c8777844b45b1a1c78282920927. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:08:44.617126 5400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98a09c8777844b45b1a1c78282920927. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:08:44.622994 5483 tablet_bootstrap.cc:492] T 98a09c8777844b45b1a1c78282920927 P 24d8419acc004656b215237292bd6388: Bootstrap starting.
I20260924 04:08:44.624572 5485 tablet_bootstrap.cc:492] T 98a09c8777844b45b1a1c78282920927 P 8d79a03d13d9402a80d48b7d81a5aed6: Bootstrap starting.
I20260924 04:08:44.625468 5483 tablet_bootstrap.cc:654] T 98a09c8777844b45b1a1c78282920927 P 24d8419acc004656b215237292bd6388: Neither blocks nor log segments found. Creating new log.
I20260924 04:08:44.626190 5483 log.cc:824] T 98a09c8777844b45b1a1c78282920927 P 24d8419acc004656b215237292bd6388: Log is configured to *not* fsync() on all Append() calls
I20260924 04:08:44.626428 5485 tablet_bootstrap.cc:654] T 98a09c8777844b45b1a1c78282920927 P 8d79a03d13d9402a80d48b7d81a5aed6: Neither blocks nor log segments found. Creating new log.
I20260924 04:08:44.627081 5485 log.cc:824] T 98a09c8777844b45b1a1c78282920927 P 8d79a03d13d9402a80d48b7d81a5aed6: Log is configured to *not* fsync() on all Append() calls
I20260924 04:08:44.627655 5484 tablet_bootstrap.cc:492] T 98a09c8777844b45b1a1c78282920927 P e4fa37b9466a44f9a82d7f3a1dc5cd93: Bootstrap starting.
I20260924 04:08:44.628118 5483 tablet_bootstrap.cc:492] T 98a09c8777844b45b1a1c78282920927 P 24d8419acc004656b215237292bd6388: No bootstrap required, opened a new log
I20260924 04:08:44.628309 5483 ts_tablet_manager.cc:1397] T 98a09c8777844b45b1a1c78282920927 P 24d8419acc004656b215237292bd6388: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 04:08:44.629680 5484 tablet_bootstrap.cc:654] T 98a09c8777844b45b1a1c78282920927 P e4fa37b9466a44f9a82d7f3a1dc5cd93: Neither blocks nor log segments found. Creating new log.
I20260924 04:08:44.630096 5485 tablet_bootstrap.cc:492] T 98a09c8777844b45b1a1c78282920927 P 8d79a03d13d9402a80d48b7d81a5aed6: No bootstrap required, opened a new log
I20260924 04:08:44.630276 5485 ts_tablet_manager.cc:1397] T 98a09c8777844b45b1a1c78282920927 P 8d79a03d13d9402a80d48b7d81a5aed6: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260924 04:08:44.630599 5484 log.cc:824] T 98a09c8777844b45b1a1c78282920927 P e4fa37b9466a44f9a82d7f3a1dc5cd93: Log is configured to *not* fsync() on all Append() calls
I20260924 04:08:44.632368 5483 raft_consensus.cc:348] T 98a09c8777844b45b1a1c78282920927 P 24d8419acc004656b215237292bd6388 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4fa37b9466a44f9a82d7f3a1dc5cd93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32781 } } peers { permanent_uuid: "8d79a03d13d9402a80d48b7d81a5aed6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33629 } } peers { permanent_uuid: "24d8419acc004656b215237292bd6388" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41511 } }
I20260924 04:08:44.632611 5483 raft_consensus.cc:374] T 98a09c8777844b45b1a1c78282920927 P 24d8419acc004656b215237292bd6388 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:08:44.632630 5484 tablet_bootstrap.cc:492] T 98a09c8777844b45b1a1c78282920927 P e4fa37b9466a44f9a82d7f3a1dc5cd93: No bootstrap required, opened a new log
I20260924 04:08:44.632666 5483 raft_consensus.cc:729] T 98a09c8777844b45b1a1c78282920927 P 24d8419acc004656b215237292bd6388 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24d8419acc004656b215237292bd6388, State: Initialized, Role: FOLLOWER
I20260924 04:08:44.632848 5484 ts_tablet_manager.cc:1397] T 98a09c8777844b45b1a1c78282920927 P e4fa37b9466a44f9a82d7f3a1dc5cd93: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260924 04:08:44.633145 5483 consensus_queue.cc:260] T 98a09c8777844b45b1a1c78282920927 P 24d8419acc004656b215237292bd6388 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4fa37b9466a44f9a82d7f3a1dc5cd93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32781 } } peers { permanent_uuid: "8d79a03d13d9402a80d48b7d81a5aed6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33629 } } peers { permanent_uuid: "24d8419acc004656b215237292bd6388" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41511 } }
I20260924 04:08:44.633782 5204 heartbeater.cc:503] Master 127.0.98.62:38647 was elected leader, sending a full tablet report...
I20260924 04:08:44.634570 5485 raft_consensus.cc:348] T 98a09c8777844b45b1a1c78282920927 P 8d79a03d13d9402a80d48b7d81a5aed6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4fa37b9466a44f9a82d7f3a1dc5cd93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32781 } } peers { permanent_uuid: "8d79a03d13d9402a80d48b7d81a5aed6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33629 } } peers { permanent_uuid: "24d8419acc004656b215237292bd6388" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41511 } }
I20260924 04:08:44.634820 5485 raft_consensus.cc:374] T 98a09c8777844b45b1a1c78282920927 P 8d79a03d13d9402a80d48b7d81a5aed6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:08:44.634876 5485 raft_consensus.cc:729] T 98a09c8777844b45b1a1c78282920927 P 8d79a03d13d9402a80d48b7d81a5aed6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d79a03d13d9402a80d48b7d81a5aed6, State: Initialized, Role: FOLLOWER
I20260924 04:08:44.635108 5483 ts_tablet_manager.cc:1428] T 98a09c8777844b45b1a1c78282920927 P 24d8419acc004656b215237292bd6388: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260924 04:08:44.635327 5485 consensus_queue.cc:260] T 98a09c8777844b45b1a1c78282920927 P 8d79a03d13d9402a80d48b7d81a5aed6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4fa37b9466a44f9a82d7f3a1dc5cd93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32781 } } peers { permanent_uuid: "8d79a03d13d9402a80d48b7d81a5aed6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33629 } } peers { permanent_uuid: "24d8419acc004656b215237292bd6388" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41511 } }
I20260924 04:08:44.636023 5335 heartbeater.cc:503] Master 127.0.98.62:38647 was elected leader, sending a full tablet report...
I20260924 04:08:44.636353 5485 ts_tablet_manager.cc:1428] T 98a09c8777844b45b1a1c78282920927 P 8d79a03d13d9402a80d48b7d81a5aed6: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260924 04:08:44.637251 5484 raft_consensus.cc:348] T 98a09c8777844b45b1a1c78282920927 P e4fa37b9466a44f9a82d7f3a1dc5cd93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4fa37b9466a44f9a82d7f3a1dc5cd93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32781 } } peers { permanent_uuid: "8d79a03d13d9402a80d48b7d81a5aed6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33629 } } peers { permanent_uuid: "24d8419acc004656b215237292bd6388" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41511 } }
I20260924 04:08:44.637531 5484 raft_consensus.cc:374] T 98a09c8777844b45b1a1c78282920927 P e4fa37b9466a44f9a82d7f3a1dc5cd93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:08:44.637583 5484 raft_consensus.cc:729] T 98a09c8777844b45b1a1c78282920927 P e4fa37b9466a44f9a82d7f3a1dc5cd93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4fa37b9466a44f9a82d7f3a1dc5cd93, State: Initialized, Role: FOLLOWER
I20260924 04:08:44.637989 5484 consensus_queue.cc:260] T 98a09c8777844b45b1a1c78282920927 P e4fa37b9466a44f9a82d7f3a1dc5cd93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4fa37b9466a44f9a82d7f3a1dc5cd93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32781 } } peers { permanent_uuid: "8d79a03d13d9402a80d48b7d81a5aed6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33629 } } peers { permanent_uuid: "24d8419acc004656b215237292bd6388" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41511 } }
I20260924 04:08:44.638676 5466 heartbeater.cc:503] Master 127.0.98.62:38647 was elected leader, sending a full tablet report...
I20260924 04:08:44.639855 5484 ts_tablet_manager.cc:1428] T 98a09c8777844b45b1a1c78282920927 P e4fa37b9466a44f9a82d7f3a1dc5cd93: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
W20260924 04:08:44.645190 5336 tablet.cc:2369] T 98a09c8777844b45b1a1c78282920927 P 8d79a03d13d9402a80d48b7d81a5aed6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 04:08:44.716951 5205 tablet.cc:2369] T 98a09c8777844b45b1a1c78282920927 P 24d8419acc004656b215237292bd6388: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:08:44.735742 5489 raft_consensus.cc:482] T 98a09c8777844b45b1a1c78282920927 P 24d8419acc004656b215237292bd6388 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:08:44.735940 5489 raft_consensus.cc:504] T 98a09c8777844b45b1a1c78282920927 P 24d8419acc004656b215237292bd6388 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4fa37b9466a44f9a82d7f3a1dc5cd93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32781 } } peers { permanent_uuid: "8d79a03d13d9402a80d48b7d81a5aed6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33629 } } peers { permanent_uuid: "24d8419acc004656b215237292bd6388" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41511 } }
I20260924 04:08:44.737254 5489 leader_election.cc:288] T 98a09c8777844b45b1a1c78282920927 P 24d8419acc004656b215237292bd6388 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4fa37b9466a44f9a82d7f3a1dc5cd93 (127.0.98.3:32781), 8d79a03d13d9402a80d48b7d81a5aed6 (127.0.98.2:33629)
I20260924 04:08:44.739620 5493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:44.737545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33629 (local address 127.0.98.1:50333)
0924 04:08:44.737871 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:44.737882 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:44.738051 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:44.738359 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:44.738364 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:44.738380 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:08:44.738906 (+ 526us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:08:44.738909 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:08:44.738922 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:08:44.738959 (+ 37us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:08:44.739251 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:08:44.739254 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:08:44.739257 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 04:08:44.739346 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 04:08:44.739390 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":87,"threads_started":1}
I20260924 04:08:44.739619 5480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:44.737644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50333 (local address 127.0.98.2:33629)
0924 04:08:44.737881 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:44.737891 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:44.737977 (+ 86us) server_negotiation.cc:407] Connection header received
0924 04:08:44.738191 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:44.738198 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:44.738243 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:44.738315 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:44.739099 (+ 784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:44.739102 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:44.739104 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:08:44.739117 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:44.739137 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:08:44.739146 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:44.739248 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:44.739354 (+ 106us) server_negotiation.cc:299] Negotiation successful
0924 04:08:44.739413 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260924 04:08:44.739652 5479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:44.737644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39299 (local address 127.0.98.3:32781)
0924 04:08:44.737778 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:44.737782 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:44.738129 (+ 347us) server_negotiation.cc:407] Connection header received
0924 04:08:44.738241 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:44.738243 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:44.738276 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:44.738374 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:44.739273 (+ 899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:44.739277 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:44.739283 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:08:44.739299 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:44.739324 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:08:44.739334 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:44.739415 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:44.739498 (+ 83us) server_negotiation.cc:299] Negotiation successful
0924 04:08:44.739535 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 04:08:44.739679 5492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:44.737561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32781 (local address 127.0.98.1:39299)
0924 04:08:44.738030 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:44.738042 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:44.738153 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:44.738423 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:44.738426 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:44.738438 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:08:44.739116 (+ 678us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:08:44.739118 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:08:44.739127 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:08:44.739151 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:08:44.739418 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:08:44.739421 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:08:44.739424 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 04:08:44.739506 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 04:08:44.739540 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":85,"threads_started":1}
I20260924 04:08:44.740469 5289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98a09c8777844b45b1a1c78282920927" candidate_uuid: "24d8419acc004656b215237292bd6388" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d79a03d13d9402a80d48b7d81a5aed6" is_pre_election: true
I20260924 04:08:44.740576 5420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98a09c8777844b45b1a1c78282920927" candidate_uuid: "24d8419acc004656b215237292bd6388" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4fa37b9466a44f9a82d7f3a1dc5cd93" is_pre_election: true
I20260924 04:08:44.740850 5289 raft_consensus.cc:2454] T 98a09c8777844b45b1a1c78282920927 P 8d79a03d13d9402a80d48b7d81a5aed6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24d8419acc004656b215237292bd6388 in term 0.
I20260924 04:08:44.740890 5420 raft_consensus.cc:2454] T 98a09c8777844b45b1a1c78282920927 P e4fa37b9466a44f9a82d7f3a1dc5cd93 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24d8419acc004656b215237292bd6388 in term 0.
I20260924 04:08:44.741423 5092 leader_election.cc:302] T 98a09c8777844b45b1a1c78282920927 P 24d8419acc004656b215237292bd6388 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 24d8419acc004656b215237292bd6388, 8d79a03d13d9402a80d48b7d81a5aed6, e4fa37b9466a44f9a82d7f3a1dc5cd93; no voters:
I20260924 04:08:44.741710 5489 raft_consensus.cc:2781] T 98a09c8777844b45b1a1c78282920927 P 24d8419acc004656b215237292bd6388 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:08:44.741796 5489 raft_consensus.cc:482] T 98a09c8777844b45b1a1c78282920927 P 24d8419acc004656b215237292bd6388 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:08:44.741838 5489 raft_consensus.cc:3037] T 98a09c8777844b45b1a1c78282920927 P 24d8419acc004656b215237292bd6388 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:08:44.742868 5489 raft_consensus.cc:504] T 98a09c8777844b45b1a1c78282920927 P 24d8419acc004656b215237292bd6388 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4fa37b9466a44f9a82d7f3a1dc5cd93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32781 } } peers { permanent_uuid: "8d79a03d13d9402a80d48b7d81a5aed6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33629 } } peers { permanent_uuid: "24d8419acc004656b215237292bd6388" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41511 } }
I20260924 04:08:44.743223 5489 leader_election.cc:288] T 98a09c8777844b45b1a1c78282920927 P 24d8419acc004656b215237292bd6388 [CANDIDATE]: Term 1 election: Requested vote from peers e4fa37b9466a44f9a82d7f3a1dc5cd93 (127.0.98.3:32781), 8d79a03d13d9402a80d48b7d81a5aed6 (127.0.98.2:33629)
I20260924 04:08:44.743583 5420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98a09c8777844b45b1a1c78282920927" candidate_uuid: "24d8419acc004656b215237292bd6388" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4fa37b9466a44f9a82d7f3a1dc5cd93"
I20260924 04:08:44.743588 5289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98a09c8777844b45b1a1c78282920927" candidate_uuid: "24d8419acc004656b215237292bd6388" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d79a03d13d9402a80d48b7d81a5aed6"
I20260924 04:08:44.743714 5289 raft_consensus.cc:3037] T 98a09c8777844b45b1a1c78282920927 P 8d79a03d13d9402a80d48b7d81a5aed6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:08:44.743714 5420 raft_consensus.cc:3037] T 98a09c8777844b45b1a1c78282920927 P e4fa37b9466a44f9a82d7f3a1dc5cd93 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:08:44.745011 5289 raft_consensus.cc:2454] T 98a09c8777844b45b1a1c78282920927 P 8d79a03d13d9402a80d48b7d81a5aed6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24d8419acc004656b215237292bd6388 in term 1.
I20260924 04:08:44.745011 5420 raft_consensus.cc:2454] T 98a09c8777844b45b1a1c78282920927 P e4fa37b9466a44f9a82d7f3a1dc5cd93 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24d8419acc004656b215237292bd6388 in term 1.
I20260924 04:08:44.745298 5094 leader_election.cc:302] T 98a09c8777844b45b1a1c78282920927 P 24d8419acc004656b215237292bd6388 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24d8419acc004656b215237292bd6388, 8d79a03d13d9402a80d48b7d81a5aed6; no voters:
I20260924 04:08:44.745512 5489 raft_consensus.cc:2781] T 98a09c8777844b45b1a1c78282920927 P 24d8419acc004656b215237292bd6388 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:08:44.745771 5489 raft_consensus.cc:686] T 98a09c8777844b45b1a1c78282920927 P 24d8419acc004656b215237292bd6388 [term 1 LEADER]: Becoming Leader. State: Replica: 24d8419acc004656b215237292bd6388, State: Running, Role: LEADER
I20260924 04:08:44.746130 5489 consensus_queue.cc:237] T 98a09c8777844b45b1a1c78282920927 P 24d8419acc004656b215237292bd6388 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4fa37b9466a44f9a82d7f3a1dc5cd93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32781 } } peers { permanent_uuid: "8d79a03d13d9402a80d48b7d81a5aed6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33629 } } peers { permanent_uuid: "24d8419acc004656b215237292bd6388" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41511 } }
I20260924 04:08:44.749468 5013 catalog_manager.cc:5495] T 98a09c8777844b45b1a1c78282920927 P 24d8419acc004656b215237292bd6388 reported cstate change: term changed from 0 to 1, leader changed from <none> to 24d8419acc004656b215237292bd6388 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "24d8419acc004656b215237292bd6388" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4fa37b9466a44f9a82d7f3a1dc5cd93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d79a03d13d9402a80d48b7d81a5aed6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24d8419acc004656b215237292bd6388" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41511 } health_report { overall_health: HEALTHY } } }
I20260924 04:08:44.765305 5482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:44.763721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35508 (local address 127.0.98.1:41511)
0924 04:08:44.763872 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:44.763878 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:44.763894 (+ 16us) server_negotiation.cc:407] Connection header received
0924 04:08:44.763982 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:44.763984 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:44.764044 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:44.764146 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:44.764894 (+ 748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:44.764897 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:44.764900 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:08:44.764912 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:44.764940 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:08:44.764948 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:44.765042 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:44.765126 (+ 84us) server_negotiation.cc:299] Negotiation successful
0924 04:08:44.765167 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 04:08:44.771492 5420 raft_consensus.cc:1261] T 98a09c8777844b45b1a1c78282920927 P e4fa37b9466a44f9a82d7f3a1dc5cd93 [term 1 FOLLOWER]: Refusing update from remote peer 24d8419acc004656b215237292bd6388: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:08:44.771492 5289 raft_consensus.cc:1261] T 98a09c8777844b45b1a1c78282920927 P 8d79a03d13d9402a80d48b7d81a5aed6 [term 1 FOLLOWER]: Refusing update from remote peer 24d8419acc004656b215237292bd6388: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:08:44.772296 5494 consensus_queue.cc:1035] T 98a09c8777844b45b1a1c78282920927 P 24d8419acc004656b215237292bd6388 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d79a03d13d9402a80d48b7d81a5aed6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:08:44.772503 5489 consensus_queue.cc:1035] T 98a09c8777844b45b1a1c78282920927 P 24d8419acc004656b215237292bd6388 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4fa37b9466a44f9a82d7f3a1dc5cd93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:08:44.780975 5497 mvcc.cc:205] Tried to move back new op lower bound from 7332753099855716352 to 7332753099762647040. Current Snapshot: MvccSnapshot[applied={T|T < 7332753099855716352 or (T in {7332753099855716352})}]
I20260924 04:08:44.781276 5499 mvcc.cc:205] Tried to move back new op lower bound from 7332753099855716352 to 7332753099762647040. Current Snapshot: MvccSnapshot[applied={T|T < 7332753099855716352}]
I20260924 04:08:44.781446 5498 mvcc.cc:205] Tried to move back new op lower bound from 7332753099855716352 to 7332753099762647040. Current Snapshot: MvccSnapshot[applied={T|T < 7332753099855716352}]
I20260924 04:08:44.790176 5016 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43744:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:08:44.790349 5016 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43744:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:08:44.792738 5016 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0814d00729fd4711b3bedb85216c831c: Sending DeleteTablet for 3 replicas of tablet 98a09c8777844b45b1a1c78282920927
I20260924 04:08:44.793497 5269 tablet_service.cc:1514] Processing DeleteTablet for tablet 98a09c8777844b45b1a1c78282920927 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:08:44 UTC) from {username='slave'} at 127.0.0.1:40972
I20260924 04:08:44.793497 5138 tablet_service.cc:1514] Processing DeleteTablet for tablet 98a09c8777844b45b1a1c78282920927 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:08:44 UTC) from {username='slave'} at 127.0.0.1:35506
I20260924 04:08:44.793550 5400 tablet_service.cc:1514] Processing DeleteTablet for tablet 98a09c8777844b45b1a1c78282920927 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:08:44 UTC) from {username='slave'} at 127.0.0.1:53188
I20260924 04:08:44.794008 5508 tablet_replica.cc:330] T 98a09c8777844b45b1a1c78282920927 P e4fa37b9466a44f9a82d7f3a1dc5cd93: stopping tablet replica
I20260924 04:08:44.794080 5506 tablet_replica.cc:330] T 98a09c8777844b45b1a1c78282920927 P 8d79a03d13d9402a80d48b7d81a5aed6: stopping tablet replica
I20260924 04:08:44.794368 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 5076
I20260924 04:08:44.794404 5506 raft_consensus.cc:2229] T 98a09c8777844b45b1a1c78282920927 P 8d79a03d13d9402a80d48b7d81a5aed6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:08:44.795436 5506 raft_consensus.cc:2258] T 98a09c8777844b45b1a1c78282920927 P 8d79a03d13d9402a80d48b7d81a5aed6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:08:44.795476 5508 raft_consensus.cc:2229] T 98a09c8777844b45b1a1c78282920927 P e4fa37b9466a44f9a82d7f3a1dc5cd93 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:08:44.796235 5508 raft_consensus.cc:2258] T 98a09c8777844b45b1a1c78282920927 P e4fa37b9466a44f9a82d7f3a1dc5cd93 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:08:44.796797 5506 ts_tablet_manager.cc:1905] T 98a09c8777844b45b1a1c78282920927 P 8d79a03d13d9402a80d48b7d81a5aed6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:08:44.797700 5508 ts_tablet_manager.cc:1905] T 98a09c8777844b45b1a1c78282920927 P e4fa37b9466a44f9a82d7f3a1dc5cd93: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:08:44.798878 5506 ts_tablet_manager.cc:1918] T 98a09c8777844b45b1a1c78282920927 P 8d79a03d13d9402a80d48b7d81a5aed6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:08:44.798956 5506 log.cc:1196] T 98a09c8777844b45b1a1c78282920927 P 8d79a03d13d9402a80d48b7d81a5aed6: Deleting WAL directory at /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222898807720-392-0/minicluster-data/ts-1/wal/wals/98a09c8777844b45b1a1c78282920927
I20260924 04:08:44.799183 5506 ts_tablet_manager.cc:1939] T 98a09c8777844b45b1a1c78282920927 P 8d79a03d13d9402a80d48b7d81a5aed6: Deleting consensus metadata
I20260924 04:08:44.800411 5003 catalog_manager.cc:4841] TS 8d79a03d13d9402a80d48b7d81a5aed6 (127.0.98.2:33629): tablet 98a09c8777844b45b1a1c78282920927 (table test-table [id=3e9cdb497482420e8a6ed7399b6278bf]) successfully deleted
I20260924 04:08:44.801465 5508 ts_tablet_manager.cc:1918] T 98a09c8777844b45b1a1c78282920927 P e4fa37b9466a44f9a82d7f3a1dc5cd93: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:08:44.801570 5508 log.cc:1196] T 98a09c8777844b45b1a1c78282920927 P e4fa37b9466a44f9a82d7f3a1dc5cd93: Deleting WAL directory at /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790222898807720-392-0/minicluster-data/ts-2/wal/wals/98a09c8777844b45b1a1c78282920927
I20260924 04:08:44.801882 5508 ts_tablet_manager.cc:1939] T 98a09c8777844b45b1a1c78282920927 P e4fa37b9466a44f9a82d7f3a1dc5cd93: Deleting consensus metadata
W20260924 04:08:44.802604 5001 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41511 (error 108)
I20260924 04:08:44.802883 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 5207
I20260924 04:08:44.803107 5001 catalog_manager.cc:4841] TS e4fa37b9466a44f9a82d7f3a1dc5cd93 (127.0.98.3:32781): tablet 98a09c8777844b45b1a1c78282920927 (table test-table [id=3e9cdb497482420e8a6ed7399b6278bf]) successfully deleted
I20260924 04:08:44.804272 5478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:44.803979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41511 (local address 127.0.0.1:35516)
0924 04:08:44.804110 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:44.804193 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260924 04:08:44.804418 5001 catalog_manager.cc:4584] TS 24d8419acc004656b215237292bd6388 (127.0.98.1:41511): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 98a09c8777844b45b1a1c78282920927: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41511: connect: Connection refused (error 111)
I20260924 04:08:44.808391 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 5338
I20260924 04:08:44.813637 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 4985
2026-09-24T04:08:44Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1066 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260924 04:08:44.835084 392 mini_oidc.cc:152] Starting JWKS server
I20260924 04:08:44.836670 392 webserver.cc:460] Webserver started at https://127.0.0.1:43371/ using document root <none> and password file <none>
I20260924 04:08:44.836906 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 04:08:44.837198 392 webserver.cc:460] Webserver started at http://127.0.0.1:39259/ using document root <none> and password file <none>
2026-09-24T04:08:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:08:44Z Disabled control of system clock
I20260924 04:08:44.846398 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790222898807720-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790222898807720-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790222898807720-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35349
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:42603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35349
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43371/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790222898807720-392-0/testchainca.cert with env {}
W20260924 04:08:44.955443 5515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:08:44.955741 5515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:08:44.955783 5515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:08:44.959805 5515 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:08:44.959872 5515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:08:44.959887 5515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:08:44.959903 5515 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:08:44.959918 5515 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 04:08:44.959934 5515 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 04:08:44.959949 5515 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43371/jwks.json
I20260924 04:08:44.965441 5515 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790222898807720-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790222898807720-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35349
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35349
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43371/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790222898807720-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5515
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790222898807720-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:08:44.966490 5515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:08:44.967509 5515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:08:44.975867 5521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:08:44.975914 5523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:08:44.975924 5515 server_base.cc:1034] running on GCE node
W20260924 04:08:44.975873 5520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:08:44.976756 5515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:08:44.977846 5515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:08:44.979017 5515 hybrid_clock.cc:648] HybridClock initialized: now 1790222924978986 us; error 47 us; skew 500 ppm
I20260924 04:08:44.981074 5515 webserver.cc:460] Webserver started at http://127.0.98.62:38297/ using document root <none> and password file <none>
I20260924 04:08:44.981709 5515 fs_manager.cc:359] Metadata directory not provided
I20260924 04:08:44.981771 5515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:08:44.981984 5515 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:08:44.984058 5515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790222898807720-392-0/minicluster-data/master-0/data/instance:
uuid: "5938bead19d146de8a6aad6bc3c76918"
format_stamp: "Formatted at 2026-09-24 04:08:44 on dist-test-slave-00ld"
server_key: "0cb3c2611c3408b174ef27c3e20379c4"
server_key_iv: "9b838ff80cc01dbedd2bcc9bede727c0"
server_key_version: "encryptionkey@0"
I20260924 04:08:44.984640 5515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790222898807720-392-0/minicluster-data/master-0/wal/instance:
uuid: "5938bead19d146de8a6aad6bc3c76918"
format_stamp: "Formatted at 2026-09-24 04:08:44 on dist-test-slave-00ld"
server_key: "0cb3c2611c3408b174ef27c3e20379c4"
server_key_iv: "9b838ff80cc01dbedd2bcc9bede727c0"
server_key_version: "encryptionkey@0"
I20260924 04:08:44.988180 5515 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:08:44.990562 5529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:44.991807 5515 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:08:44.991945 5515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790222898807720-392-0/minicluster-data/master-0/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790222898807720-392-0/minicluster-data/master-0/wal
uuid: "5938bead19d146de8a6aad6bc3c76918"
format_stamp: "Formatted at 2026-09-24 04:08:44 on dist-test-slave-00ld"
server_key: "0cb3c2611c3408b174ef27c3e20379c4"
server_key_iv: "9b838ff80cc01dbedd2bcc9bede727c0"
server_key_version: "encryptionkey@0"
I20260924 04:08:44.992051 5515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790222898807720-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790222898807720-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790222898807720-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:08:45.006996 5515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:08:45.007649 5515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:08:45.007848 5515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:08:45.014667 5515 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35349
I20260924 04:08:45.014683 5581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35349 every 8 connection(s)
I20260924 04:08:45.015978 5515 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:08:45.019258 5582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:08:45.022235 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 5515
I20260924 04:08:45.022327 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790222898807720-392-0/minicluster-data/master-0/wal/instance
I20260924 04:08:45.022617 392 external_mini_cluster.cc:1428] Setting key 2699e84b361e229b5ec50de9c82953ee
I20260924 04:08:45.025264 5582 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5938bead19d146de8a6aad6bc3c76918: Bootstrap starting.
I20260924 04:08:45.028029 5582 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5938bead19d146de8a6aad6bc3c76918: Neither blocks nor log segments found. Creating new log.
I20260924 04:08:45.028951 5582 log.cc:824] T 00000000000000000000000000000000 P 5938bead19d146de8a6aad6bc3c76918: Log is configured to *not* fsync() on all Append() calls
I20260924 04:08:45.031184 5582 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5938bead19d146de8a6aad6bc3c76918: No bootstrap required, opened a new log
I20260924 04:08:45.033282 5585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:45.024028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40230 (local address 127.0.98.62:35349)
0924 04:08:45.024728 (+ 700us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:45.024748 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:45.024793 (+ 45us) server_negotiation.cc:407] Connection header received
0924 04:08:45.025607 (+ 814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:45.025632 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:45.026133 (+ 501us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:45.026521 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:45.026983 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:45.028452 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:45.029576 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:45.030213 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:45.030367 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:45.031196 (+ 829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:45.031205 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:45.031223 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:08:45.031731 (+ 508us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:45.031800 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:08:45.031825 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:45.031917 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:45.032069 (+ 152us) server_negotiation.cc:299] Negotiation successful
0924 04:08:45.032327 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":103,"threads_started":1}
I20260924 04:08:45.035652 5582 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5938bead19d146de8a6aad6bc3c76918 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5938bead19d146de8a6aad6bc3c76918" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35349 } }
I20260924 04:08:45.035892 5582 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5938bead19d146de8a6aad6bc3c76918 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:08:45.035959 5582 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5938bead19d146de8a6aad6bc3c76918 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5938bead19d146de8a6aad6bc3c76918, State: Initialized, Role: FOLLOWER
I20260924 04:08:45.036067 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:08:45.036425 5582 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5938bead19d146de8a6aad6bc3c76918 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5938bead19d146de8a6aad6bc3c76918" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35349 } }
I20260924 04:08:45.036544 5582 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5938bead19d146de8a6aad6bc3c76918 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:08:45.036626 5582 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5938bead19d146de8a6aad6bc3c76918 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:08:45.036772 5582 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5938bead19d146de8a6aad6bc3c76918 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:08:45.037878 5582 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5938bead19d146de8a6aad6bc3c76918 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5938bead19d146de8a6aad6bc3c76918" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35349 } }
I20260924 04:08:45.038215 5582 leader_election.cc:302] T 00000000000000000000000000000000 P 5938bead19d146de8a6aad6bc3c76918 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5938bead19d146de8a6aad6bc3c76918; no voters:
I20260924 04:08:45.038514 5582 leader_election.cc:288] T 00000000000000000000000000000000 P 5938bead19d146de8a6aad6bc3c76918 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:08:45.038688 5587 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5938bead19d146de8a6aad6bc3c76918 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:08:45.038918 5587 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5938bead19d146de8a6aad6bc3c76918 [term 1 LEADER]: Becoming Leader. State: Replica: 5938bead19d146de8a6aad6bc3c76918, State: Running, Role: LEADER
I20260924 04:08:45.039280 5587 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5938bead19d146de8a6aad6bc3c76918 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5938bead19d146de8a6aad6bc3c76918" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35349 } }
I20260924 04:08:45.039775 5582 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5938bead19d146de8a6aad6bc3c76918 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:08:45.040997 5588 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5938bead19d146de8a6aad6bc3c76918 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5938bead19d146de8a6aad6bc3c76918" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5938bead19d146de8a6aad6bc3c76918" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35349 } } }
I20260924 04:08:45.041144 5588 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5938bead19d146de8a6aad6bc3c76918 [sys.catalog]: This master's current role is: LEADER
I20260924 04:08:45.041544 5596 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:08:45.041486 5589 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5938bead19d146de8a6aad6bc3c76918 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5938bead19d146de8a6aad6bc3c76918. Latest consensus state: current_term: 1 leader_uuid: "5938bead19d146de8a6aad6bc3c76918" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5938bead19d146de8a6aad6bc3c76918" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35349 } } }
I20260924 04:08:45.041644 5589 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5938bead19d146de8a6aad6bc3c76918 [sys.catalog]: This master's current role is: LEADER
I20260924 04:08:45.044463 5596 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:08:45.049746 5596 catalog_manager.cc:1347] Generated new cluster ID: d860885903224dd8b50ebe7b9fd75654
I20260924 04:08:45.049824 5596 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:08:45.059648 5596 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:08:45.060657 5596 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:08:45.067191 5596 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5938bead19d146de8a6aad6bc3c76918: Generated new TSK 0
I20260924 04:08:45.067977 5596 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 04:08:45.104903 5612 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260924 04:08:45.105022 5612 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 04:08:45.105510 5585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:45.080771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40246 (local address 127.0.98.62:35349)
0924 04:08:45.080919 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:45.080924 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:45.081166 (+ 242us) server_negotiation.cc:407] Connection header received
0924 04:08:45.081401 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:45.081404 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:45.081451 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:45.081547 (+ 96us) server_negotiation.cc:226] Negotiated authn=JWT
0924 04:08:45.081821 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:45.082368 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:45.083599 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:45.083822 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:45.083904 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:45.084270 (+ 366us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 04:08:45.105115 (+ 20845us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43371/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 04:08:45.105376 (+ 261us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40246: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43371/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":60}
W20260924 04:08:45.105618 5585 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40246: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43371/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260924 04:08:45.106904 5611 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:08:45.080712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35349 (local address 127.0.0.1:40246)
0924 04:08:45.081070 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:45.081087 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:45.081232 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:45.081576 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:45.081580 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:45.081588 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0924 04:08:45.081712 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:45.081719 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:45.082479 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:45.082485 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:45.083472 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:45.083482 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:45.083919 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:45.083940 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:45.084028 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:45.084080 (+ 52us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 04:08:45.105366 (+ 21286us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43371/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 04:08:45.105608 (+ 242us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:35349: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43371/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":113,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:35349: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43371/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260924 04:08:45.107921 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 5515
2026-09-24T04:08:45Z chronyd exiting
I20260924 04:08:45.145474 392 test_util.cc:175] -----------------------------------------------
I20260924 04:08:45.145609 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790222898807720-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (312 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260924 04:08:45.146660 392 mini_oidc.cc:152] Starting JWKS server
I20260924 04:08:45.147471 392 webserver.cc:460] Webserver started at https://127.0.0.1:36945/ using document root <none> and password file <none>
I20260924 04:08:45.147559 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 04:08:45.147761 392 webserver.cc:460] Webserver started at http://127.0.0.1:37361/ using document root <none> and password file <none>
2026-09-24T04:08:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:08:45Z Disabled control of system clock
I20260924 04:08:45.157222 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790222898807720-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790222898807720-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790222898807720-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44331
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:36461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44331
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36945/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790222898807720-392-0/testchainca.cert with env {}
W20260924 04:08:45.265842 5618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:08:45.266110 5618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:08:45.266150 5618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:08:45.270068 5618 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:08:45.270123 5618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:08:45.270139 5618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:08:45.270155 5618 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:08:45.270170 5618 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 04:08:45.270186 5618 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 04:08:45.270200 5618 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36945/jwks.json
I20260924 04:08:45.275472 5618 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790222898807720-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790222898807720-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44331
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36945/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790222898807720-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5618
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790222898807720-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:08:45.276475 5618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:08:45.277550 5618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:08:45.285863 5623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:08:45.285941 5618 server_base.cc:1034] running on GCE node
W20260924 04:08:45.285857 5624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:08:45.285867 5626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:08:45.286679 5618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:08:45.287734 5618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:08:45.288887 5618 hybrid_clock.cc:648] HybridClock initialized: now 1790222925288876 us; error 24 us; skew 500 ppm
I20260924 04:08:45.290642 5618 webserver.cc:460] Webserver started at http://127.0.98.62:35443/ using document root <none> and password file <none>
I20260924 04:08:45.291205 5618 fs_manager.cc:359] Metadata directory not provided
I20260924 04:08:45.291266 5618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:08:45.291507 5618 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:08:45.293648 5618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790222898807720-392-0/minicluster-data/master-0/data/instance:
uuid: "29d3875f8ad34bdf8ee1f65d525afe27"
format_stamp: "Formatted at 2026-09-24 04:08:45 on dist-test-slave-00ld"
server_key: "f8d76d76f1f6bc5eb6ee5a91195cb735"
server_key_iv: "bfef33bce43d0cf8ede4810612db56bd"
server_key_version: "encryptionkey@0"
I20260924 04:08:45.294250 5618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790222898807720-392-0/minicluster-data/master-0/wal/instance:
uuid: "29d3875f8ad34bdf8ee1f65d525afe27"
format_stamp: "Formatted at 2026-09-24 04:08:45 on dist-test-slave-00ld"
server_key: "f8d76d76f1f6bc5eb6ee5a91195cb735"
server_key_iv: "bfef33bce43d0cf8ede4810612db56bd"
server_key_version: "encryptionkey@0"
I20260924 04:08:45.297930 5618 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:08:45.300274 5632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:45.301569 5618 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:08:45.301712 5618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790222898807720-392-0/minicluster-data/master-0/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790222898807720-392-0/minicluster-data/master-0/wal
uuid: "29d3875f8ad34bdf8ee1f65d525afe27"
format_stamp: "Formatted at 2026-09-24 04:08:45 on dist-test-slave-00ld"
server_key: "f8d76d76f1f6bc5eb6ee5a91195cb735"
server_key_iv: "bfef33bce43d0cf8ede4810612db56bd"
server_key_version: "encryptionkey@0"
I20260924 04:08:45.301826 5618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790222898807720-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790222898807720-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790222898807720-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:08:45.318333 5618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:08:45.319073 5618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:08:45.319285 5618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:08:45.326099 5618 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44331
I20260924 04:08:45.326112 5684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44331 every 8 connection(s)
I20260924 04:08:45.327052 5618 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:08:45.330240 5685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:08:45.332752 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 5618
I20260924 04:08:45.332878 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790222898807720-392-0/minicluster-data/master-0/wal/instance
I20260924 04:08:45.333184 392 external_mini_cluster.cc:1428] Setting key d2fd475cdbdc96749cc470bb33769d1f
I20260924 04:08:45.336367 5685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29d3875f8ad34bdf8ee1f65d525afe27: Bootstrap starting.
I20260924 04:08:45.338835 5685 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 29d3875f8ad34bdf8ee1f65d525afe27: Neither blocks nor log segments found. Creating new log.
I20260924 04:08:45.339705 5685 log.cc:824] T 00000000000000000000000000000000 P 29d3875f8ad34bdf8ee1f65d525afe27: Log is configured to *not* fsync() on all Append() calls
I20260924 04:08:45.341996 5685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29d3875f8ad34bdf8ee1f65d525afe27: No bootstrap required, opened a new log
I20260924 04:08:45.343371 5688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:45.334554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47756 (local address 127.0.98.62:44331)
0924 04:08:45.335196 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:45.335210 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:45.335244 (+ 34us) server_negotiation.cc:407] Connection header received
0924 04:08:45.335867 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:45.335882 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:45.336278 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:45.336618 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:45.337029 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:45.338004 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:45.339221 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:45.339905 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:45.340094 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:45.341224 (+ 1130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:45.341250 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:45.341266 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:08:45.341750 (+ 484us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:45.341817 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:08:45.341842 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:45.341937 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:45.342209 (+ 272us) server_negotiation.cc:299] Negotiation successful
0924 04:08:45.342478 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":131,"threads_started":1}
I20260924 04:08:45.346060 5685 raft_consensus.cc:348] T 00000000000000000000000000000000 P 29d3875f8ad34bdf8ee1f65d525afe27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29d3875f8ad34bdf8ee1f65d525afe27" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44331 } }
I20260924 04:08:45.346359 5685 raft_consensus.cc:374] T 00000000000000000000000000000000 P 29d3875f8ad34bdf8ee1f65d525afe27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:08:45.346424 5685 raft_consensus.cc:729] T 00000000000000000000000000000000 P 29d3875f8ad34bdf8ee1f65d525afe27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d3875f8ad34bdf8ee1f65d525afe27, State: Initialized, Role: FOLLOWER
I20260924 04:08:45.346902 5685 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29d3875f8ad34bdf8ee1f65d525afe27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29d3875f8ad34bdf8ee1f65d525afe27" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44331 } }
I20260924 04:08:45.347007 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:08:45.347074 5685 raft_consensus.cc:388] T 00000000000000000000000000000000 P 29d3875f8ad34bdf8ee1f65d525afe27 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:08:45.347136 5685 raft_consensus.cc:482] T 00000000000000000000000000000000 P 29d3875f8ad34bdf8ee1f65d525afe27 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:08:45.347224 5685 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29d3875f8ad34bdf8ee1f65d525afe27 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:08:45.348392 5685 raft_consensus.cc:504] T 00000000000000000000000000000000 P 29d3875f8ad34bdf8ee1f65d525afe27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29d3875f8ad34bdf8ee1f65d525afe27" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44331 } }
I20260924 04:08:45.348791 5685 leader_election.cc:302] T 00000000000000000000000000000000 P 29d3875f8ad34bdf8ee1f65d525afe27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 29d3875f8ad34bdf8ee1f65d525afe27; no voters:
I20260924 04:08:45.349123 5685 leader_election.cc:288] T 00000000000000000000000000000000 P 29d3875f8ad34bdf8ee1f65d525afe27 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:08:45.349227 5690 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 29d3875f8ad34bdf8ee1f65d525afe27 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:08:45.349458 5690 raft_consensus.cc:686] T 00000000000000000000000000000000 P 29d3875f8ad34bdf8ee1f65d525afe27 [term 1 LEADER]: Becoming Leader. State: Replica: 29d3875f8ad34bdf8ee1f65d525afe27, State: Running, Role: LEADER
I20260924 04:08:45.349774 5690 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29d3875f8ad34bdf8ee1f65d525afe27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29d3875f8ad34bdf8ee1f65d525afe27" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44331 } }
I20260924 04:08:45.350464 5685 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29d3875f8ad34bdf8ee1f65d525afe27 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:08:45.351560 5691 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29d3875f8ad34bdf8ee1f65d525afe27 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "29d3875f8ad34bdf8ee1f65d525afe27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29d3875f8ad34bdf8ee1f65d525afe27" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44331 } } }
I20260924 04:08:45.351694 5691 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29d3875f8ad34bdf8ee1f65d525afe27 [sys.catalog]: This master's current role is: LEADER
I20260924 04:08:45.351603 5692 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29d3875f8ad34bdf8ee1f65d525afe27 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29d3875f8ad34bdf8ee1f65d525afe27. Latest consensus state: current_term: 1 leader_uuid: "29d3875f8ad34bdf8ee1f65d525afe27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29d3875f8ad34bdf8ee1f65d525afe27" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44331 } } }
I20260924 04:08:45.351851 5692 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29d3875f8ad34bdf8ee1f65d525afe27 [sys.catalog]: This master's current role is: LEADER
I20260924 04:08:45.352185 5699 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:08:45.355247 5699 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:08:45.360263 5699 catalog_manager.cc:1347] Generated new cluster ID: c94a8a8d2d464bcd959f9008aea85b0c
I20260924 04:08:45.360329 5699 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:08:45.390919 5699 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:08:45.391861 5699 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:08:45.403872 5699 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 29d3875f8ad34bdf8ee1f65d525afe27: Generated new TSK 0
I20260924 04:08:45.404778 5699 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 04:08:45.432962 5715 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260924 04:08:45.433063 5715 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 04:08:45.433813 5688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:45.418100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47772 (local address 127.0.98.62:44331)
0924 04:08:45.418230 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:45.418236 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:45.418396 (+ 160us) server_negotiation.cc:407] Connection header received
0924 04:08:45.418514 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:45.418517 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:45.418572 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:45.418684 (+ 112us) server_negotiation.cc:226] Negotiated authn=JWT
0924 04:08:45.418876 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:45.419690 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:45.420536 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:45.420907 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:45.421008 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:45.421262 (+ 254us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 04:08:45.433376 (+ 12114us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36945/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 04:08:45.433668 (+ 292us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47772: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36945/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":53}
W20260924 04:08:45.433881 5688 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47772: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36945/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260924 04:08:45.433933 5714 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:08:45.417990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44331 (local address 127.0.0.1:47772)
0924 04:08:45.418305 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:45.418317 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:45.418412 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:45.418686 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:45.418688 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:45.418694 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0924 04:08:45.418780 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:45.418785 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:45.419818 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:45.419820 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:45.420426 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:45.420432 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:45.420998 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:45.421013 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:45.421078 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:45.421099 (+ 21us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 04:08:45.433587 (+ 12488us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36945/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 04:08:45.433759 (+ 172us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:44331: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36945/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":133,"threads_started":1}
I20260924 04:08:45.434341 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 5618
2026-09-24T04:08:45Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (326 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260924 04:08:45.472783 392 mini_oidc.cc:152] Starting JWKS server
I20260924 04:08:45.473647 392 webserver.cc:460] Webserver started at https://127.0.0.1:39239/ using document root <none> and password file <none>
I20260924 04:08:45.473778 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 04:08:45.474032 392 webserver.cc:460] Webserver started at http://127.0.0.1:33829/ using document root <none> and password file <none>
2026-09-24T04:08:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:08:45Z Disabled control of system clock
I20260924 04:08:45.482826 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790222898807720-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790222898807720-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790222898807720-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45763
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:42793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45763
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39239/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 04:08:45.591845 5721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:08:45.592120 5721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:08:45.592159 5721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:08:45.596169 5721 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:08:45.596227 5721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:08:45.596242 5721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:08:45.596259 5721 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:08:45.596274 5721 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 04:08:45.596289 5721 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 04:08:45.596303 5721 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39239/jwks.json
I20260924 04:08:45.601121 5721 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790222898807720-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790222898807720-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45763
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45763
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39239/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5721
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790222898807720-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:08:45.602100 5721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:08:45.603425 5721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:08:45.611742 5726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:08:45.611737 5729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:08:45.611764 5727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:08:45.612026 5721 server_base.cc:1034] running on GCE node
I20260924 04:08:45.612583 5721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:08:45.613672 5721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:08:45.614858 5721 hybrid_clock.cc:648] HybridClock initialized: now 1790222925614842 us; error 26 us; skew 500 ppm
I20260924 04:08:45.616835 5721 webserver.cc:460] Webserver started at http://127.0.98.62:44509/ using document root <none> and password file <none>
I20260924 04:08:45.617398 5721 fs_manager.cc:359] Metadata directory not provided
I20260924 04:08:45.617476 5721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:08:45.617739 5721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:08:45.619724 5721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790222898807720-392-0/minicluster-data/master-0/data/instance:
uuid: "3ce3963d28fb4bdca9f8b82d32a56c7e"
format_stamp: "Formatted at 2026-09-24 04:08:45 on dist-test-slave-00ld"
server_key: "f80e0a42437c3226d1481fd4a47f5ce0"
server_key_iv: "3e80de7c8f63651b6e4a7ecc9097f035"
server_key_version: "encryptionkey@0"
I20260924 04:08:45.620296 5721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790222898807720-392-0/minicluster-data/master-0/wal/instance:
uuid: "3ce3963d28fb4bdca9f8b82d32a56c7e"
format_stamp: "Formatted at 2026-09-24 04:08:45 on dist-test-slave-00ld"
server_key: "f80e0a42437c3226d1481fd4a47f5ce0"
server_key_iv: "3e80de7c8f63651b6e4a7ecc9097f035"
server_key_version: "encryptionkey@0"
I20260924 04:08:45.623848 5721 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260924 04:08:45.626246 5735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:45.627362 5721 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:08:45.627501 5721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790222898807720-392-0/minicluster-data/master-0/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790222898807720-392-0/minicluster-data/master-0/wal
uuid: "3ce3963d28fb4bdca9f8b82d32a56c7e"
format_stamp: "Formatted at 2026-09-24 04:08:45 on dist-test-slave-00ld"
server_key: "f80e0a42437c3226d1481fd4a47f5ce0"
server_key_iv: "3e80de7c8f63651b6e4a7ecc9097f035"
server_key_version: "encryptionkey@0"
I20260924 04:08:45.627612 5721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790222898807720-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790222898807720-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790222898807720-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:08:45.645851 5721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:08:45.646587 5721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:08:45.646796 5721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:08:45.653508 5721 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45763
I20260924 04:08:45.653532 5787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45763 every 8 connection(s)
I20260924 04:08:45.654527 5721 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:08:45.657853 5788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:08:45.658399 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 5721
I20260924 04:08:45.658521 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790222898807720-392-0/minicluster-data/master-0/wal/instance
I20260924 04:08:45.658838 392 external_mini_cluster.cc:1428] Setting key d22420686956180cfb6235fe8e5576ca
I20260924 04:08:45.664146 5788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ce3963d28fb4bdca9f8b82d32a56c7e: Bootstrap starting.
I20260924 04:08:45.666441 5788 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ce3963d28fb4bdca9f8b82d32a56c7e: Neither blocks nor log segments found. Creating new log.
I20260924 04:08:45.667165 5788 log.cc:824] T 00000000000000000000000000000000 P 3ce3963d28fb4bdca9f8b82d32a56c7e: Log is configured to *not* fsync() on all Append() calls
I20260924 04:08:45.668264 5791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:45.660269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46638 (local address 127.0.98.62:45763)
0924 04:08:45.660978 (+ 709us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:45.660994 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:45.661029 (+ 35us) server_negotiation.cc:407] Connection header received
0924 04:08:45.661722 (+ 693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:45.661739 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:45.662148 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:45.662495 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:45.662918 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:45.663891 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:45.664949 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:45.665425 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:45.665625 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:45.666490 (+ 865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:45.666495 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:45.666509 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:08:45.666964 (+ 455us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:45.667019 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:08:45.667040 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:45.667149 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:45.667266 (+ 117us) server_negotiation.cc:299] Negotiation successful
0924 04:08:45.667534 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":131,"threads_started":1}
I20260924 04:08:45.669376 5788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ce3963d28fb4bdca9f8b82d32a56c7e: No bootstrap required, opened a new log
I20260924 04:08:45.671283 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:08:45.673758 5788 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ce3963d28fb4bdca9f8b82d32a56c7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ce3963d28fb4bdca9f8b82d32a56c7e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45763 } }
I20260924 04:08:45.673977 5788 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3ce3963d28fb4bdca9f8b82d32a56c7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:08:45.674062 5788 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ce3963d28fb4bdca9f8b82d32a56c7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ce3963d28fb4bdca9f8b82d32a56c7e, State: Initialized, Role: FOLLOWER
I20260924 04:08:45.674580 5788 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ce3963d28fb4bdca9f8b82d32a56c7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ce3963d28fb4bdca9f8b82d32a56c7e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45763 } }
I20260924 04:08:45.674768 5788 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3ce3963d28fb4bdca9f8b82d32a56c7e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:08:45.674830 5788 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3ce3963d28fb4bdca9f8b82d32a56c7e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:08:45.674932 5788 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ce3963d28fb4bdca9f8b82d32a56c7e [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:08:45.675854 5788 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3ce3963d28fb4bdca9f8b82d32a56c7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ce3963d28fb4bdca9f8b82d32a56c7e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45763 } }
I20260924 04:08:45.676209 5788 leader_election.cc:302] T 00000000000000000000000000000000 P 3ce3963d28fb4bdca9f8b82d32a56c7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3ce3963d28fb4bdca9f8b82d32a56c7e; no voters:
I20260924 04:08:45.676532 5788 leader_election.cc:288] T 00000000000000000000000000000000 P 3ce3963d28fb4bdca9f8b82d32a56c7e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:08:45.676766 5793 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3ce3963d28fb4bdca9f8b82d32a56c7e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:08:45.677022 5793 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3ce3963d28fb4bdca9f8b82d32a56c7e [term 1 LEADER]: Becoming Leader. State: Replica: 3ce3963d28fb4bdca9f8b82d32a56c7e, State: Running, Role: LEADER
I20260924 04:08:45.677352 5793 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3ce3963d28fb4bdca9f8b82d32a56c7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ce3963d28fb4bdca9f8b82d32a56c7e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45763 } }
I20260924 04:08:45.677865 5788 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ce3963d28fb4bdca9f8b82d32a56c7e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:08:45.679082 5795 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ce3963d28fb4bdca9f8b82d32a56c7e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ce3963d28fb4bdca9f8b82d32a56c7e. Latest consensus state: current_term: 1 leader_uuid: "3ce3963d28fb4bdca9f8b82d32a56c7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ce3963d28fb4bdca9f8b82d32a56c7e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45763 } } }
I20260924 04:08:45.679232 5795 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ce3963d28fb4bdca9f8b82d32a56c7e [sys.catalog]: This master's current role is: LEADER
I20260924 04:08:45.679826 5794 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ce3963d28fb4bdca9f8b82d32a56c7e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3ce3963d28fb4bdca9f8b82d32a56c7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ce3963d28fb4bdca9f8b82d32a56c7e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45763 } } }
I20260924 04:08:45.680485 5794 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ce3963d28fb4bdca9f8b82d32a56c7e [sys.catalog]: This master's current role is: LEADER
W20260924 04:08:45.682691 5808 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3ce3963d28fb4bdca9f8b82d32a56c7e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:08:45.682780 5808 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:08:45.682855 5802 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:08:45.683506 5802 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:08:45.688629 5802 catalog_manager.cc:1347] Generated new cluster ID: f6e84da346d14bcaba3610f65055978e
I20260924 04:08:45.688741 5802 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:08:45.695526 5802 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:08:45.696482 5802 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:08:45.705061 5802 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3ce3963d28fb4bdca9f8b82d32a56c7e: Generated new TSK 0
I20260924 04:08:45.705760 5802 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:08:45.723469 5791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:45.718871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46654 (local address 127.0.98.62:45763)
0924 04:08:45.718991 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:45.718997 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:45.719194 (+ 197us) server_negotiation.cc:407] Connection header received
0924 04:08:45.719390 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:45.719392 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:45.719437 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:45.719552 (+ 115us) server_negotiation.cc:226] Negotiated authn=JWT
0924 04:08:45.719784 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:45.720320 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:45.721276 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:45.721502 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:45.721617 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:45.721924 (+ 307us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 04:08:45.723082 (+ 1158us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39239/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 04:08:45.723319 (+ 237us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46654: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39239/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":42}
W20260924 04:08:45.723532 5817 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:08:45.718800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45763 (local address 127.0.0.1:46654)
0924 04:08:45.719105 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:45.719118 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:45.719244 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:45.719553 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:45.719556 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:45.719564 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0924 04:08:45.719672 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:45.719678 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:45.720437 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:45.720441 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:45.721158 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:45.721167 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:45.721610 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:45.721630 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:45.721726 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:45.721748 (+ 22us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 04:08:45.723242 (+ 1494us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39239/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 04:08:45.723376 (+ 134us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:45763: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39239/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":104,"threads_started":1}
I20260924 04:08:45.724009 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 5721
W20260924 04:08:45.724023 5791 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46654: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39239/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
E20260924 04:08:45.734336 5818 webserver.cc:533] Webserver: sslize failed:
E20260924 04:08:45.734407 5818 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-24T04:08:45Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (290 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-24T04:08:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:08:45Z Disabled control of system clock
I20260924 04:08:45.772212 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790222898807720-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790222898807720-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790222898807720-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39185
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:36105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39185
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790222898807720-392-0/testchainca.cert with env {}
W20260924 04:08:45.883528 5822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:08:45.883821 5822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:08:45.883863 5822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:08:45.887938 5822 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:08:45.888000 5822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:08:45.888015 5822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:08:45.888033 5822 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:08:45.888048 5822 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 04:08:45.888063 5822 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 04:08:45.888077 5822 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260924 04:08:45.893426 5822 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790222898807720-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790222898807720-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39185
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39185
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790222898807720-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5822
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790222898807720-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:08:45.894412 5822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:08:45.895350 5822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:08:45.903787 5827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:08:45.903770 5830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:08:45.903803 5828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:08:45.903877 5822 server_base.cc:1034] running on GCE node
I20260924 04:08:45.904590 5822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:08:45.905874 5822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:08:45.907058 5822 hybrid_clock.cc:648] HybridClock initialized: now 1790222925907039 us; error 29 us; skew 500 ppm
I20260924 04:08:45.908964 5822 webserver.cc:460] Webserver started at http://127.0.98.62:40581/ using document root <none> and password file <none>
I20260924 04:08:45.909559 5822 fs_manager.cc:359] Metadata directory not provided
I20260924 04:08:45.909626 5822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:08:45.909883 5822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:08:45.911887 5822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790222898807720-392-0/minicluster-data/master-0/data/instance:
uuid: "185e07521ca846ecb00cd22dc8e678cc"
format_stamp: "Formatted at 2026-09-24 04:08:45 on dist-test-slave-00ld"
server_key: "ee65eb6eda16a14f2bda4a499586c74a"
server_key_iv: "56106edff03aea9598d13794e34b5f2f"
server_key_version: "encryptionkey@0"
I20260924 04:08:45.912456 5822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790222898807720-392-0/minicluster-data/master-0/wal/instance:
uuid: "185e07521ca846ecb00cd22dc8e678cc"
format_stamp: "Formatted at 2026-09-24 04:08:45 on dist-test-slave-00ld"
server_key: "ee65eb6eda16a14f2bda4a499586c74a"
server_key_iv: "56106edff03aea9598d13794e34b5f2f"
server_key_version: "encryptionkey@0"
I20260924 04:08:45.916021 5822 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260924 04:08:45.918428 5836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:45.919616 5822 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:08:45.919752 5822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790222898807720-392-0/minicluster-data/master-0/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790222898807720-392-0/minicluster-data/master-0/wal
uuid: "185e07521ca846ecb00cd22dc8e678cc"
format_stamp: "Formatted at 2026-09-24 04:08:45 on dist-test-slave-00ld"
server_key: "ee65eb6eda16a14f2bda4a499586c74a"
server_key_iv: "56106edff03aea9598d13794e34b5f2f"
server_key_version: "encryptionkey@0"
I20260924 04:08:45.919862 5822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790222898807720-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790222898807720-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790222898807720-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:08:45.932301 5822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:08:45.932999 5822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:08:45.933193 5822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:08:45.939985 5822 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39185
I20260924 04:08:45.939991 5888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39185 every 8 connection(s)
I20260924 04:08:45.940907 5822 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:08:45.943866 5889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:08:45.948076 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 5822
I20260924 04:08:45.948222 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790222898807720-392-0/minicluster-data/master-0/wal/instance
I20260924 04:08:45.948482 392 external_mini_cluster.cc:1428] Setting key c44fc144f03c8b6501f06063bfaced60
I20260924 04:08:45.949568 5889 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 185e07521ca846ecb00cd22dc8e678cc: Bootstrap starting.
I20260924 04:08:45.951742 5889 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 185e07521ca846ecb00cd22dc8e678cc: Neither blocks nor log segments found. Creating new log.
I20260924 04:08:45.952479 5889 log.cc:824] T 00000000000000000000000000000000 P 185e07521ca846ecb00cd22dc8e678cc: Log is configured to *not* fsync() on all Append() calls
I20260924 04:08:45.954319 5889 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 185e07521ca846ecb00cd22dc8e678cc: No bootstrap required, opened a new log
I20260924 04:08:45.957636 5892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:45.949959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58102 (local address 127.0.98.62:39185)
0924 04:08:45.950565 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:45.950580 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:45.950616 (+ 36us) server_negotiation.cc:407] Connection header received
0924 04:08:45.951233 (+ 617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:45.951249 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:45.951632 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:45.951960 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:45.952393 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:45.953436 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:45.954324 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:45.954851 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:45.955026 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:45.955894 (+ 868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:45.955898 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:45.955911 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:08:45.956298 (+ 387us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:45.956352 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:08:45.956371 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:45.956482 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:45.956634 (+ 152us) server_negotiation.cc:299] Negotiation successful
0924 04:08:45.956909 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":130,"threads_started":1}
I20260924 04:08:45.958163 5889 raft_consensus.cc:348] T 00000000000000000000000000000000 P 185e07521ca846ecb00cd22dc8e678cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "185e07521ca846ecb00cd22dc8e678cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39185 } }
I20260924 04:08:45.958425 5889 raft_consensus.cc:374] T 00000000000000000000000000000000 P 185e07521ca846ecb00cd22dc8e678cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:08:45.958479 5889 raft_consensus.cc:729] T 00000000000000000000000000000000 P 185e07521ca846ecb00cd22dc8e678cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 185e07521ca846ecb00cd22dc8e678cc, State: Initialized, Role: FOLLOWER
I20260924 04:08:45.958889 5889 consensus_queue.cc:260] T 00000000000000000000000000000000 P 185e07521ca846ecb00cd22dc8e678cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "185e07521ca846ecb00cd22dc8e678cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39185 } }
I20260924 04:08:45.959007 5889 raft_consensus.cc:388] T 00000000000000000000000000000000 P 185e07521ca846ecb00cd22dc8e678cc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:08:45.959093 5889 raft_consensus.cc:482] T 00000000000000000000000000000000 P 185e07521ca846ecb00cd22dc8e678cc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:08:45.959210 5889 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 185e07521ca846ecb00cd22dc8e678cc [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:08:45.960312 5889 raft_consensus.cc:504] T 00000000000000000000000000000000 P 185e07521ca846ecb00cd22dc8e678cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "185e07521ca846ecb00cd22dc8e678cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39185 } }
I20260924 04:08:45.960623 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:08:45.960713 5889 leader_election.cc:302] T 00000000000000000000000000000000 P 185e07521ca846ecb00cd22dc8e678cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 185e07521ca846ecb00cd22dc8e678cc; no voters:
I20260924 04:08:45.961027 5889 leader_election.cc:288] T 00000000000000000000000000000000 P 185e07521ca846ecb00cd22dc8e678cc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:08:45.961135 5894 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 185e07521ca846ecb00cd22dc8e678cc [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:08:45.961378 5894 raft_consensus.cc:686] T 00000000000000000000000000000000 P 185e07521ca846ecb00cd22dc8e678cc [term 1 LEADER]: Becoming Leader. State: Replica: 185e07521ca846ecb00cd22dc8e678cc, State: Running, Role: LEADER
I20260924 04:08:45.961694 5894 consensus_queue.cc:237] T 00000000000000000000000000000000 P 185e07521ca846ecb00cd22dc8e678cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "185e07521ca846ecb00cd22dc8e678cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39185 } }
I20260924 04:08:45.962306 5889 sys_catalog.cc:564] T 00000000000000000000000000000000 P 185e07521ca846ecb00cd22dc8e678cc [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:08:45.963436 5895 sys_catalog.cc:455] T 00000000000000000000000000000000 P 185e07521ca846ecb00cd22dc8e678cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "185e07521ca846ecb00cd22dc8e678cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "185e07521ca846ecb00cd22dc8e678cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39185 } } }
I20260924 04:08:45.963600 5895 sys_catalog.cc:458] T 00000000000000000000000000000000 P 185e07521ca846ecb00cd22dc8e678cc [sys.catalog]: This master's current role is: LEADER
I20260924 04:08:45.963891 5896 sys_catalog.cc:455] T 00000000000000000000000000000000 P 185e07521ca846ecb00cd22dc8e678cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 185e07521ca846ecb00cd22dc8e678cc. Latest consensus state: current_term: 1 leader_uuid: "185e07521ca846ecb00cd22dc8e678cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "185e07521ca846ecb00cd22dc8e678cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39185 } } }
I20260924 04:08:45.964025 5896 sys_catalog.cc:458] T 00000000000000000000000000000000 P 185e07521ca846ecb00cd22dc8e678cc [sys.catalog]: This master's current role is: LEADER
I20260924 04:08:45.964004 5903 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:08:45.967186 5903 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:08:45.972419 5903 catalog_manager.cc:1347] Generated new cluster ID: 4831b00152fd40b590a8ada2980039d9
I20260924 04:08:45.972486 5903 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:08:45.986151 5903 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:08:45.987118 5903 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:08:45.993122 5903 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 185e07521ca846ecb00cd22dc8e678cc: Generated new TSK 0
I20260924 04:08:45.993821 5903 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:08:46.083081 5892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:46.005509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58106 (local address 127.0.98.62:39185)
0924 04:08:46.005631 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:46.005636 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:46.005857 (+ 221us) server_negotiation.cc:407] Connection header received
0924 04:08:46.005985 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:46.005988 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:46.006053 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:46.006168 (+ 115us) server_negotiation.cc:226] Negotiated authn=JWT
0924 04:08:46.006371 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:46.007195 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:46.008044 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:46.008390 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:46.008482 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:46.008798 (+ 316us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 04:08:46.082581 (+ 73783us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 04:08:46.082924 (+ 343us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58106: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":47}
W20260924 04:08:46.083179 5892 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58106: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260924 04:08:46.083221 5918 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:08:46.005470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39185 (local address 127.0.0.1:58106)
0924 04:08:46.005776 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:46.005789 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:46.005880 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:46.006169 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:46.006172 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:46.006178 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0924 04:08:46.006278 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:46.006282 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:46.007302 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:46.007305 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:46.007937 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:46.007945 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:46.008496 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:46.008511 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:46.008579 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:46.008621 (+ 42us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 04:08:46.082864 (+ 74243us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 04:08:46.083030 (+ 166us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:39185: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":134,"threads_started":1}
I20260924 04:08:46.084007 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 5822
2026-09-24T04:08:46Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (339 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (123 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (126 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222898807720-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:08:46 dist-test-slave-00ld krb5kdc[5930](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:08:46 dist-test-slave-00ld krb5kdc[5930](info): set up 2 sockets
Sep 24 04:08:46 dist-test-slave-00ld krb5kdc[5930](info): commencing operation
krb5kdc: starting...
W20260924 04:08:47.373183 392 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222898807720-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222898807720-392-0/krb5kdc/test-user.keytab.
Sep 24 04:08:47 dist-test-slave-00ld krb5kdc[5930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222927, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:08:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:08:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:08:47.495936 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222898807720-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222898807720-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222898807720-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32891
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32891
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:08:47.606210 5943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:08:47.606482 5943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:08:47.606521 5943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:08:47.610682 5943 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:08:47.610742 5943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:08:47.610761 5943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:08:47.610778 5943 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:08:47.610793 5943 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:08:47.615990 5943 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222898807720-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222898807720-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32891
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5943
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222898807720-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:08:47.617008 5943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:08:47.617990 5943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:08:47.626546 5943 server_base.cc:1034] running on GCE node
W20260924 04:08:47.626521 5949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:08:47.626488 5951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:08:47.626490 5948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:08:47.627238 5943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:08:47.628373 5943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:08:47.629531 5943 hybrid_clock.cc:648] HybridClock initialized: now 1790222927629491 us; error 55 us; skew 500 ppm
Sep 24 04:08:47 dist-test-slave-00ld krb5kdc[5930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222927, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:08:47.632264 5943 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:08:47.633263 5943 webserver.cc:460] Webserver started at http://127.0.98.62:44255/ using document root <none> and password file <none>
I20260924 04:08:47.633845 5943 fs_manager.cc:359] Metadata directory not provided
I20260924 04:08:47.633906 5943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:08:47.634140 5943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:08:47.636138 5943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222898807720-392-0/minicluster-data/master-0/data/instance:
uuid: "24068a58107840ec8f4a53e71a10afef"
format_stamp: "Formatted at 2026-09-24 04:08:47 on dist-test-slave-00ld"
server_key: "3dee6a8ff222ea16a0ade01af34a8cf9"
server_key_iv: "c7f0f12fe8149bdfafc97de0037150f8"
server_key_version: "encryptionkey@0"
I20260924 04:08:47.636760 5943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222898807720-392-0/minicluster-data/master-0/wal/instance:
uuid: "24068a58107840ec8f4a53e71a10afef"
format_stamp: "Formatted at 2026-09-24 04:08:47 on dist-test-slave-00ld"
server_key: "3dee6a8ff222ea16a0ade01af34a8cf9"
server_key_iv: "c7f0f12fe8149bdfafc97de0037150f8"
server_key_version: "encryptionkey@0"
I20260924 04:08:47.640385 5943 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:08:47.642946 5958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:47.644263 5943 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:08:47.644402 5943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222898807720-392-0/minicluster-data/master-0/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222898807720-392-0/minicluster-data/master-0/wal
uuid: "24068a58107840ec8f4a53e71a10afef"
format_stamp: "Formatted at 2026-09-24 04:08:47 on dist-test-slave-00ld"
server_key: "3dee6a8ff222ea16a0ade01af34a8cf9"
server_key_iv: "c7f0f12fe8149bdfafc97de0037150f8"
server_key_version: "encryptionkey@0"
I20260924 04:08:47.644563 5943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222898807720-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222898807720-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222898807720-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:08:47.659927 5943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:08:47.663447 5943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:08:47.663663 5943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:08:47.670953 5943 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:32891
I20260924 04:08:47.671255 6010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:32891 every 8 connection(s)
I20260924 04:08:47.671983 5943 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:08:47.674795 6011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:08:47.680176 6011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24068a58107840ec8f4a53e71a10afef: Bootstrap starting.
I20260924 04:08:47.682215 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 5943
I20260924 04:08:47.682367 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222898807720-392-0/minicluster-data/master-0/wal/instance
I20260924 04:08:47.682628 392 external_mini_cluster.cc:1428] Setting key 17c440a5d808c03c8a87ca30d960a6d3
I20260924 04:08:47.682789 6011 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 24068a58107840ec8f4a53e71a10afef: Neither blocks nor log segments found. Creating new log.
I20260924 04:08:47.683722 6011 log.cc:824] T 00000000000000000000000000000000 P 24068a58107840ec8f4a53e71a10afef: Log is configured to *not* fsync() on all Append() calls
I20260924 04:08:47.685734 6011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24068a58107840ec8f4a53e71a10afef: No bootstrap required, opened a new log
I20260924 04:08:47.689957 6011 raft_consensus.cc:348] T 00000000000000000000000000000000 P 24068a58107840ec8f4a53e71a10afef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24068a58107840ec8f4a53e71a10afef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32891 } }
I20260924 04:08:47.690249 6011 raft_consensus.cc:374] T 00000000000000000000000000000000 P 24068a58107840ec8f4a53e71a10afef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:08:47.690320 6011 raft_consensus.cc:729] T 00000000000000000000000000000000 P 24068a58107840ec8f4a53e71a10afef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24068a58107840ec8f4a53e71a10afef, State: Initialized, Role: FOLLOWER
I20260924 04:08:47.690852 6011 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24068a58107840ec8f4a53e71a10afef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24068a58107840ec8f4a53e71a10afef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32891 } }
I20260924 04:08:47.691051 6011 raft_consensus.cc:388] T 00000000000000000000000000000000 P 24068a58107840ec8f4a53e71a10afef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:08:47.691128 6011 raft_consensus.cc:482] T 00000000000000000000000000000000 P 24068a58107840ec8f4a53e71a10afef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:08:47.691242 6011 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 24068a58107840ec8f4a53e71a10afef [term 0 FOLLOWER]: Advancing to term 1
Sep 24 04:08:47 dist-test-slave-00ld krb5kdc[5930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222927, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:08:47.692425 6011 raft_consensus.cc:504] T 00000000000000000000000000000000 P 24068a58107840ec8f4a53e71a10afef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24068a58107840ec8f4a53e71a10afef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32891 } }
I20260924 04:08:47.692819 6011 leader_election.cc:302] T 00000000000000000000000000000000 P 24068a58107840ec8f4a53e71a10afef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 24068a58107840ec8f4a53e71a10afef; no voters:
I20260924 04:08:47.693162 6011 leader_election.cc:288] T 00000000000000000000000000000000 P 24068a58107840ec8f4a53e71a10afef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:08:47.693311 6016 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 24068a58107840ec8f4a53e71a10afef [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:08:47.693528 6016 raft_consensus.cc:686] T 00000000000000000000000000000000 P 24068a58107840ec8f4a53e71a10afef [term 1 LEADER]: Becoming Leader. State: Replica: 24068a58107840ec8f4a53e71a10afef, State: Running, Role: LEADER
I20260924 04:08:47.693852 6016 consensus_queue.cc:237] T 00000000000000000000000000000000 P 24068a58107840ec8f4a53e71a10afef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24068a58107840ec8f4a53e71a10afef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32891 } }
I20260924 04:08:47.694545 6011 sys_catalog.cc:564] T 00000000000000000000000000000000 P 24068a58107840ec8f4a53e71a10afef [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:08:47.695094 6016 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24068a58107840ec8f4a53e71a10afef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "24068a58107840ec8f4a53e71a10afef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24068a58107840ec8f4a53e71a10afef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32891 } } }
I20260924 04:08:47.695206 6016 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24068a58107840ec8f4a53e71a10afef [sys.catalog]: This master's current role is: LEADER
I20260924 04:08:47.695398 6016 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24068a58107840ec8f4a53e71a10afef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24068a58107840ec8f4a53e71a10afef. Latest consensus state: current_term: 1 leader_uuid: "24068a58107840ec8f4a53e71a10afef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24068a58107840ec8f4a53e71a10afef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32891 } } }
I20260924 04:08:47.695485 6016 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24068a58107840ec8f4a53e71a10afef [sys.catalog]: This master's current role is: LEADER
I20260924 04:08:47.698493 6014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:47.683773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50722 (local address 127.0.98.62:32891)
0924 04:08:47.684230 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:47.684246 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:47.684280 (+ 34us) server_negotiation.cc:407] Connection header received
0924 04:08:47.684999 (+ 719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:47.685018 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:47.685333 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:47.685609 (+ 276us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:47.686565 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:47.687519 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:47.688776 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:47.689231 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:47.689349 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:47.692332 (+ 2983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:47.692337 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:47.692350 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:47.692377 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:47.694529 (+ 2152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:47.695200 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:47.695206 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:47.695214 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:47.695364 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:47.695796 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:47.695802 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:47.695805 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:47.696326 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:47.696538 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:47.697091 (+ 553us) server_negotiation.cc:299] Negotiation successful
0924 04:08:47.697531 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":161,"threads_started":1}
W20260924 04:08:47.700623 6031 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 24068a58107840ec8f4a53e71a10afef: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:08:47.700712 6031 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:08:47.700770 6021 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:08:47.701452 6021 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:08:47.706575 6021 catalog_manager.cc:1347] Generated new cluster ID: 91687bfa547141f49d5b3168fe750a73
I20260924 04:08:47.706645 6021 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:08:47.726595 6021 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:08:47.727924 6021 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:08:47.736624 6021 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 24068a58107840ec8f4a53e71a10afef: Generated new TSK 0
I20260924 04:08:47.737418 6021 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222898807720-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222898807720-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:08:47.768406 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222898807720-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222898807720-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222898807720-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222898807720-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32891
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222898807720-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:08:47.882211 6037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:08:47.882498 6037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:08:47.882541 6037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:08:47.886709 6037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:08:47.886772 6037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:08:47.886868 6037 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:08:47.892073 6037 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222898807720-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222898807720-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222898807720-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222898807720-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6037
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222898807720-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:08:47.893157 6037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:08:47.894328 6037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:08:47.904137 6045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:08:47.904255 6043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:08:47.904304 6037 server_base.cc:1034] running on GCE node
W20260924 04:08:47.904134 6042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:08:47.904889 6037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:08:47.905586 6037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:08:47.906744 6037 hybrid_clock.cc:648] HybridClock initialized: now 1790222927906712 us; error 45 us; skew 500 ppm
Sep 24 04:08:47 dist-test-slave-00ld krb5kdc[5930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222927, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:08:47.915793 6037 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:08:47.916766 6037 webserver.cc:460] Webserver started at http://127.0.98.1:39669/ using document root <none> and password file <none>
I20260924 04:08:47.917338 6037 fs_manager.cc:359] Metadata directory not provided
I20260924 04:08:47.917413 6037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:08:47.917675 6037 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:08:47.919970 6037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222898807720-392-0/minicluster-data/ts-0/data/instance:
uuid: "b843b9e965d04a6798a27f4dcabadf61"
format_stamp: "Formatted at 2026-09-24 04:08:47 on dist-test-slave-00ld"
server_key: "83415532312f0db0f52f8182e1998ddc"
server_key_iv: "1c11e2ec2df0ecee8e9f54b50eaf2ebb"
server_key_version: "encryptionkey@0"
I20260924 04:08:47.920575 6037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222898807720-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b843b9e965d04a6798a27f4dcabadf61"
format_stamp: "Formatted at 2026-09-24 04:08:47 on dist-test-slave-00ld"
server_key: "83415532312f0db0f52f8182e1998ddc"
server_key_iv: "1c11e2ec2df0ecee8e9f54b50eaf2ebb"
server_key_version: "encryptionkey@0"
I20260924 04:08:47.924228 6037 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 04:08:47.926797 6052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:47.928021 6037 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:08:47.928120 6037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222898807720-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222898807720-392-0/minicluster-data/ts-0/wal
uuid: "b843b9e965d04a6798a27f4dcabadf61"
format_stamp: "Formatted at 2026-09-24 04:08:47 on dist-test-slave-00ld"
server_key: "83415532312f0db0f52f8182e1998ddc"
server_key_iv: "1c11e2ec2df0ecee8e9f54b50eaf2ebb"
server_key_version: "encryptionkey@0"
I20260924 04:08:47.928253 6037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222898807720-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222898807720-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222898807720-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:08:47.961113 6037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:08:47.964665 6037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:08:47.964901 6037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:08:47.965554 6037 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:08:47.966336 6037 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:08:47.966382 6037 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:47.966420 6037 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:08:47.966435 6037 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:47.975456 6037 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36043
I20260924 04:08:47.975479 6165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36043 every 8 connection(s)
I20260924 04:08:47.976380 6037 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222898807720-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:08:47.985571 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 6037
I20260924 04:08:47.985706 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222898807720-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:08:47.985992 392 external_mini_cluster.cc:1428] Setting key a96b7f181b05279adf05aba8cbb3a7f6
Sep 24 04:08:47 dist-test-slave-00ld krb5kdc[5930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222927, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:08:47.990576 6014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:47.978274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52173 (local address 127.0.98.62:32891)
0924 04:08:47.978416 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:47.978420 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:47.979293 (+ 873us) server_negotiation.cc:407] Connection header received
0924 04:08:47.980653 (+ 1360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:47.980656 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:47.980762 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:47.980845 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:47.982031 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:47.982584 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:47.983764 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:47.984027 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:47.984118 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:47.986842 (+ 2724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:47.986846 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:47.986849 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:47.986877 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:47.988354 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:47.989037 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:47.989043 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:47.989045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:47.989150 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:47.989470 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:47.989476 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:47.989478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:47.989724 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:47.989829 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:47.990348 (+ 519us) server_negotiation.cc:299] Negotiation successful
0924 04:08:47.990447 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 04:08:47.991899 6168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:47.978668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32891 (local address 127.0.98.1:52173)
0924 04:08:47.979125 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:47.979169 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:47.980440 (+ 1271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:47.981029 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:47.981036 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:47.981392 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:08:47.981878 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:47.981892 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:47.982719 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:47.982723 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:47.983613 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:47.983622 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:47.984189 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:47.984192 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:47.984337 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:47.985001 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:08:47.985018 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:08:47.986675 (+ 1657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:08:47.988482 (+ 1807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:47.988491 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:47.988514 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:47.988910 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:47.989247 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:47.989249 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:47.989251 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:47.989382 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:47.989826 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:08:47.989831 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:08:47.990066 (+ 235us) client_negotiation.cc:769] Sending connection context
0924 04:08:47.990300 (+ 234us) client_negotiation.cc:240] Negotiation successful
0924 04:08:47.990712 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":120,"threads_started":1}
I20260924 04:08:47.993202 6166 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32891
I20260924 04:08:47.993485 6166 heartbeater.cc:464] Registering TS with master...
I20260924 04:08:47.994067 6166 heartbeater.cc:511] Master 127.0.98.62:32891 requested a full tablet report, sending...
I20260924 04:08:47.995924 5975 ts_manager.cc:194] Registered new tserver with Master: b843b9e965d04a6798a27f4dcabadf61 (127.0.98.1:36043)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 04:08:47.997272 5975 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52173
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222898807720-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222898807720-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:08:48.016217 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222898807720-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222898807720-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222898807720-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222898807720-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32891
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222898807720-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:08:48.126622 6171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:08:48.126865 6171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:08:48.126906 6171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:08:48.130873 6171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:08:48.130924 6171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:08:48.130980 6171 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:08:48.136209 6171 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222898807720-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222898807720-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222898807720-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222898807720-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6171
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222898807720-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:08:48.137151 6171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:08:48.138088 6171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:08:48.147778 6179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:08:48.147771 6177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:08:48.147951 6171 server_base.cc:1034] running on GCE node
W20260924 04:08:48.147771 6176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:08:48.148368 6171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:08:48.149050 6171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:08:48.150205 6171 hybrid_clock.cc:648] HybridClock initialized: now 1790222928150193 us; error 27 us; skew 500 ppm
Sep 24 04:08:48 dist-test-slave-00ld krb5kdc[5930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222928, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:08:48.153034 6171 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:08:48.153848 6171 webserver.cc:460] Webserver started at http://127.0.98.2:43111/ using document root <none> and password file <none>
I20260924 04:08:48.154356 6171 fs_manager.cc:359] Metadata directory not provided
I20260924 04:08:48.154414 6171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:08:48.154573 6171 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:08:48.156610 6171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222898807720-392-0/minicluster-data/ts-1/data/instance:
uuid: "edd7a3940ed945679d78a79600637555"
format_stamp: "Formatted at 2026-09-24 04:08:48 on dist-test-slave-00ld"
server_key: "9dbd9e1cd9792d7e361876d1f68fded1"
server_key_iv: "1f8ef3d5cbdc6245b7cb14c3665bf5de"
server_key_version: "encryptionkey@0"
I20260924 04:08:48.157265 6171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222898807720-392-0/minicluster-data/ts-1/wal/instance:
uuid: "edd7a3940ed945679d78a79600637555"
format_stamp: "Formatted at 2026-09-24 04:08:48 on dist-test-slave-00ld"
server_key: "9dbd9e1cd9792d7e361876d1f68fded1"
server_key_iv: "1f8ef3d5cbdc6245b7cb14c3665bf5de"
server_key_version: "encryptionkey@0"
I20260924 04:08:48.161249 6171 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 04:08:48.163743 6186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:48.165220 6171 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:08:48.165361 6171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222898807720-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222898807720-392-0/minicluster-data/ts-1/wal
uuid: "edd7a3940ed945679d78a79600637555"
format_stamp: "Formatted at 2026-09-24 04:08:48 on dist-test-slave-00ld"
server_key: "9dbd9e1cd9792d7e361876d1f68fded1"
server_key_iv: "1f8ef3d5cbdc6245b7cb14c3665bf5de"
server_key_version: "encryptionkey@0"
I20260924 04:08:48.165482 6171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222898807720-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222898807720-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222898807720-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:08:48.185508 6171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:08:48.188633 6171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:08:48.188876 6171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:08:48.189482 6171 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:08:48.190317 6171 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:08:48.190387 6171 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:48.190465 6171 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:08:48.190517 6171 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:48.200402 6171 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33387
I20260924 04:08:48.200419 6299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33387 every 8 connection(s)
I20260924 04:08:48.201445 6171 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222898807720-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:08:48.202337 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 6171
I20260924 04:08:48.202459 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222898807720-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:08:48.202754 392 external_mini_cluster.cc:1428] Setting key b797b436f35307541c325cfbdca5f4fb
Sep 24 04:08:48 dist-test-slave-00ld krb5kdc[5930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222928, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260924 04:08:48.216744 6014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:48.203338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50331 (local address 127.0.98.62:32891)
0924 04:08:48.203489 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:48.203493 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:48.204487 (+ 994us) server_negotiation.cc:407] Connection header received
0924 04:08:48.205913 (+ 1426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:48.205916 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:48.205993 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:48.206070 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:48.207430 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:48.207958 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:48.209991 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:48.210201 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:48.210283 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:48.213137 (+ 2854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:48.213140 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:48.213142 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:48.213164 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:48.214638 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:48.215146 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:48.215149 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:48.215151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:48.215229 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:48.215574 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:48.215577 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:48.215583 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:48.215805 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:48.215908 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:48.216402 (+ 494us) server_negotiation.cc:299] Negotiation successful
0924 04:08:48.216527 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 04:08:48.217876 6302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:48.203806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32891 (local address 127.0.98.2:50331)
0924 04:08:48.204340 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:48.204383 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:48.205704 (+ 1321us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:48.206241 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:48.206249 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:48.206602 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:08:48.207265 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:48.207287 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:48.208087 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:48.208094 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:48.209866 (+ 1772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:48.209879 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:48.210356 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:48.210362 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:48.210595 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:48.211215 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:08:48.211239 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:08:48.212947 (+ 1708us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:08:48.214766 (+ 1819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:48.214771 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:48.214785 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:48.215039 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:48.215320 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:48.215324 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:48.215327 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:48.215467 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:48.215904 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:08:48.215910 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:08:48.216163 (+ 253us) client_negotiation.cc:769] Sending connection context
0924 04:08:48.216349 (+ 186us) client_negotiation.cc:240] Negotiation successful
0924 04:08:48.216710 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":156,"threads_started":1}
I20260924 04:08:48.219105 6300 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32891
I20260924 04:08:48.219363 6300 heartbeater.cc:464] Registering TS with master...
I20260924 04:08:48.219854 6300 heartbeater.cc:511] Master 127.0.98.62:32891 requested a full tablet report, sending...
I20260924 04:08:48.220938 5975 ts_manager.cc:194] Registered new tserver with Master: edd7a3940ed945679d78a79600637555 (127.0.98.2:33387)
I20260924 04:08:48.221514 5975 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50331
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222898807720-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222898807720-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:08:48.232187 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222898807720-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222898807720-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222898807720-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222898807720-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32891
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222898807720-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:08:48.347425 6305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:08:48.347697 6305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:08:48.347779 6305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:08:48.352008 6305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:08:48.352077 6305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:08:48.352136 6305 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:08:48.357439 6305 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222898807720-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222898807720-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222898807720-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222898807720-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6305
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222898807720-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:08:48.358493 6305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:08:48.359442 6305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:08:48.369563 6305 server_base.cc:1034] running on GCE node
W20260924 04:08:48.369583 6310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:08:48.369632 6313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:08:48.369601 6311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:08:48.370261 6305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:08:48.370918 6305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:08:48.372112 6305 hybrid_clock.cc:648] HybridClock initialized: now 1790222928372079 us; error 48 us; skew 500 ppm
Sep 24 04:08:48 dist-test-slave-00ld krb5kdc[5930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222928, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:08:48.374799 6305 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:08:48.375697 6305 webserver.cc:460] Webserver started at http://127.0.98.3:41069/ using document root <none> and password file <none>
I20260924 04:08:48.376267 6305 fs_manager.cc:359] Metadata directory not provided
I20260924 04:08:48.376354 6305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:08:48.376580 6305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:08:48.378643 6305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222898807720-392-0/minicluster-data/ts-2/data/instance:
uuid: "34385e4257ff45408bca8178cba8399f"
format_stamp: "Formatted at 2026-09-24 04:08:48 on dist-test-slave-00ld"
server_key: "4bd1a4102e75b118055541e13bd522c8"
server_key_iv: "dcfba9a26f1d67131b0c0d4dcd839e66"
server_key_version: "encryptionkey@0"
I20260924 04:08:48.379200 6305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222898807720-392-0/minicluster-data/ts-2/wal/instance:
uuid: "34385e4257ff45408bca8178cba8399f"
format_stamp: "Formatted at 2026-09-24 04:08:48 on dist-test-slave-00ld"
server_key: "4bd1a4102e75b118055541e13bd522c8"
server_key_iv: "dcfba9a26f1d67131b0c0d4dcd839e66"
server_key_version: "encryptionkey@0"
I20260924 04:08:48.382869 6305 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 04:08:48.385456 6320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:48.387095 6305 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:08:48.387233 6305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222898807720-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222898807720-392-0/minicluster-data/ts-2/wal
uuid: "34385e4257ff45408bca8178cba8399f"
format_stamp: "Formatted at 2026-09-24 04:08:48 on dist-test-slave-00ld"
server_key: "4bd1a4102e75b118055541e13bd522c8"
server_key_iv: "dcfba9a26f1d67131b0c0d4dcd839e66"
server_key_version: "encryptionkey@0"
I20260924 04:08:48.387369 6305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222898807720-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222898807720-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222898807720-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:08:48.422191 6305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:08:48.425513 6305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:08:48.425671 6305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:08:48.426251 6305 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:08:48.427112 6305 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:08:48.427157 6305 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:48.427197 6305 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:08:48.427212 6305 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:48.436472 6305 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42989
I20260924 04:08:48.436489 6433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42989 every 8 connection(s)
I20260924 04:08:48.437512 6305 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222898807720-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:08:48.439195 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 6305
I20260924 04:08:48.439363 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222898807720-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:08:48.439702 392 external_mini_cluster.cc:1428] Setting key 61fb8e3a045f9b322f7f6bcb11ff08e2
Sep 24 04:08:48 dist-test-slave-00ld krb5kdc[5930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222928, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:08:48.451535 6014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:48.439485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53363 (local address 127.0.98.62:32891)
0924 04:08:48.439650 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:48.439655 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:48.440608 (+ 953us) server_negotiation.cc:407] Connection header received
0924 04:08:48.442016 (+ 1408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:48.442022 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:48.442077 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:48.442157 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:48.443312 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:48.444104 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:48.445306 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:48.445535 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:48.445631 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:48.448051 (+ 2420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:48.448054 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:48.448056 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:48.448080 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:48.449689 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:48.450157 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:48.450159 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:48.450160 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:48.450231 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:48.450553 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:48.450555 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:48.450556 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:48.450734 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:48.450807 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:48.451291 (+ 484us) server_negotiation.cc:299] Negotiation successful
0924 04:08:48.451405 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 04:08:48.452483 6436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:48.439976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32891 (local address 127.0.98.3:53363)
0924 04:08:48.440443 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:48.440497 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:48.441813 (+ 1316us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:48.442305 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:48.442311 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:48.442667 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:08:48.443165 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:48.443179 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:48.444223 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:48.444226 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:48.445191 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:48.445200 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:48.445671 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:48.445673 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:48.445814 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:48.446355 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:08:48.446370 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:08:48.447847 (+ 1477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:08:48.449793 (+ 1946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:48.449798 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:48.449811 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:48.450070 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:48.450307 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:48.450309 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:48.450311 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:48.450458 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:48.450826 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:08:48.450832 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:08:48.451088 (+ 256us) client_negotiation.cc:769] Sending connection context
0924 04:08:48.451249 (+ 161us) client_negotiation.cc:240] Negotiation successful
0924 04:08:48.451504 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":136,"threads_started":1}
I20260924 04:08:48.453779 6434 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32891
I20260924 04:08:48.454041 6434 heartbeater.cc:464] Registering TS with master...
I20260924 04:08:48.454528 6434 heartbeater.cc:511] Master 127.0.98.62:32891 requested a full tablet report, sending...
I20260924 04:08:48.455538 5975 ts_manager.cc:194] Registered new tserver with Master: 34385e4257ff45408bca8178cba8399f (127.0.98.3:42989)
I20260924 04:08:48.456117 5975 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53363
I20260924 04:08:48.466082 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222898807720-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222898807720-392-0/krb5kdc/test-admin.keytab.
Sep 24 04:08:48 dist-test-slave-00ld krb5kdc[5930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:08:48.482285 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260924 04:08:48.485424 6441 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:08:48.483769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56256 (local address 127.0.98.1:36043)
0924 04:08:48.484102 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:48.484118 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:48.484142 (+ 24us) server_negotiation.cc:407] Connection header received
0924 04:08:48.484199 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:48.484205 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:48.484404 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:48.484560 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:48.485144 (+ 584us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56256: BlockingRecv error: recv got EOF from 127.0.0.1:56256 (error 108)
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":125,"threads_started":1}
W20260924 04:08:48.485549 392 init.cc:244] Error while opening credential cache '/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222898807720-392-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222898807720-392-0/krb5kdc/krb5cc)
Sep 24 04:08:48 dist-test-slave-00ld krb5kdc[5930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:08:48.486611 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 04:08:48 dist-test-slave-00ld krb5kdc[5930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:08:48.487552 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 04:08:48 dist-test-slave-00ld krb5kdc[5930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 04:08:48.497766 6441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:48.488432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56260 (local address 127.0.98.1:36043)
0924 04:08:48.488583 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:48.488586 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:48.488600 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:08:48.488748 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:48.488751 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:48.488791 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:48.488859 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:48.489345 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:48.490199 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:48.491100 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:48.491494 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:48.491580 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:48.494092 (+ 2512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:48.494102 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:48.494114 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:48.494140 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:48.496003 (+ 1863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:48.496452 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:48.496459 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:48.496461 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:48.496551 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:48.496838 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:48.496841 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:48.496842 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:48.497102 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:48.497252 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:48.497428 (+ 176us) server_negotiation.cc:299] Negotiation successful
0924 04:08:48.497597 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260924 04:08:48.501667 6441 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:08:48.500623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56266 (local address 127.0.98.1:36043)
0924 04:08:48.500784 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:48.500787 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:48.500833 (+ 46us) server_negotiation.cc:407] Connection header received
0924 04:08:48.500934 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:48.500936 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:48.500974 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:48.501038 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:48.501546 (+ 508us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56266: BlockingRecv error: recv got EOF from 127.0.0.1:56266 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260924 04:08:48.501679 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222898807720-392-0/krb5kdc/krb5cc' of type FILE
Sep 24 04:08:48 dist-test-slave-00ld krb5kdc[5930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:08:48.502717 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 04:08:48 dist-test-slave-00ld krb5kdc[5930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:08:48.503561 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 04:08:48 dist-test-slave-00ld krb5kdc[5930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 04:08:48.512573 6441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:48.504312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56274 (local address 127.0.98.1:36043)
0924 04:08:48.504439 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:48.504442 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:48.504456 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:08:48.504580 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:48.504582 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:48.504614 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:48.504719 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:48.505198 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:48.505678 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:48.506446 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:48.506664 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:48.506746 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:48.509393 (+ 2647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:48.509396 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:48.509398 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:48.509420 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:48.510988 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:48.511393 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:48.511396 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:48.511397 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:48.511469 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:48.511731 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:48.511734 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:48.511735 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:48.511902 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:48.512031 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:48.512277 (+ 246us) server_negotiation.cc:299] Negotiation successful
0924 04:08:48.512373 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260924 04:08:48.515566 6441 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:08:48.514557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56288 (local address 127.0.98.1:36043)
0924 04:08:48.514675 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:48.514680 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:48.514769 (+ 89us) server_negotiation.cc:407] Connection header received
0924 04:08:48.514927 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:48.514930 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:48.514970 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:48.515081 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:48.515464 (+ 383us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56288: BlockingRecv error: recv got EOF from 127.0.0.1:56288 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260924 04:08:48.515841 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790222898807720-392-0/krb5kdc/krb5cc' of type FILE
Sep 24 04:08:48 dist-test-slave-00ld krb5kdc[5930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:08:48.517028 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 04:08:48 dist-test-slave-00ld krb5kdc[5930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:08:48.518098 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 04:08:48 dist-test-slave-00ld krb5kdc[5930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 04:08:48.526638 6441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:48.518984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56294 (local address 127.0.98.1:36043)
0924 04:08:48.519114 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:48.519118 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:48.519131 (+ 13us) server_negotiation.cc:407] Connection header received
0924 04:08:48.519229 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:48.519231 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:48.519263 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:48.519327 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:48.519790 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:48.520280 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:48.521118 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:48.521329 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:48.521408 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:48.523646 (+ 2238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:48.523649 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:48.523651 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:48.523672 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:48.524968 (+ 1296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:48.525479 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:48.525482 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:48.525483 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:48.525562 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:48.525813 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:48.525815 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:48.525816 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:48.525972 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:48.526088 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:48.526305 (+ 217us) server_negotiation.cc:299] Negotiation successful
0924 04:08:48.526426 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 04:08:48.527861 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 6037
I20260924 04:08:48.533516 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 6171
I20260924 04:08:48.539150 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 6305
I20260924 04:08:48.544747 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 5943
2026-09-24T04:08:48Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2213 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:08:48 dist-test-slave-00ld krb5kdc[6454](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:08:48 dist-test-slave-00ld krb5kdc[6454](info): set up 2 sockets
Sep 24 04:08:48 dist-test-slave-00ld krb5kdc[6454](info): commencing operation
krb5kdc: starting...
W20260924 04:08:49.624660 392 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260924 04:08:49.871112 392 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.246s user 0.001s sys 0.000s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260924 04:08:50.189329 392 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.318s user 0.002s sys 0.000s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/krb5kdc/test-user.keytab.
Sep 24 04:08:50 dist-test-slave-00ld krb5kdc[6454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222930, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:08:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:08:50Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260924 04:08:50.271739 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38949
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38949
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:08:50.382284 6467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:08:50.382560 6467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:08:50.382602 6467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:08:50.386583 6467 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:08:50.386646 6467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:08:50.386665 6467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:08:50.386682 6467 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:08:50.386698 6467 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:08:50.392107 6467 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38949
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38949
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6467
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:08:50.393186 6467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:08:50.394143 6467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:08:50.402525 6475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:08:50.402514 6473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:08:50.402513 6472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:08:50.402678 6467 server_base.cc:1034] running on GCE node
I20260924 04:08:50.403755 6467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:08:50.405323 6467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:08:50.406539 6467 hybrid_clock.cc:648] HybridClock initialized: now 1790222930406515 us; error 36 us; skew 500 ppm
Sep 24 04:08:50 dist-test-slave-00ld krb5kdc[6454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222930, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:08:50.410424 6467 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260924 04:08:50.411700 6467 webserver.cc:460] Webserver started at http://127.0.98.62:42701/ using document root <none> and password file <none>
I20260924 04:08:50.412531 6467 fs_manager.cc:359] Metadata directory not provided
I20260924 04:08:50.412648 6467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:08:50.413008 6467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:08:50.416044 6467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/master-0/data/instance:
uuid: "cd568f5711864cc4a75a0f2fef7236a2"
format_stamp: "Formatted at 2026-09-24 04:08:50 on dist-test-slave-00ld"
server_key: "b589ee91f458b49b86cf087f00d8220c"
server_key_iv: "53abfc3f406c81bf863fd4c1a895a4e9"
server_key_version: "encryptionkey@0"
I20260924 04:08:50.416934 6467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/master-0/wal/instance:
uuid: "cd568f5711864cc4a75a0f2fef7236a2"
format_stamp: "Formatted at 2026-09-24 04:08:50 on dist-test-slave-00ld"
server_key: "b589ee91f458b49b86cf087f00d8220c"
server_key_iv: "53abfc3f406c81bf863fd4c1a895a4e9"
server_key_version: "encryptionkey@0"
I20260924 04:08:50.422130 6467 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260924 04:08:50.425228 6482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:50.426630 6467 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:08:50.426784 6467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/master-0/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/master-0/wal
uuid: "cd568f5711864cc4a75a0f2fef7236a2"
format_stamp: "Formatted at 2026-09-24 04:08:50 on dist-test-slave-00ld"
server_key: "b589ee91f458b49b86cf087f00d8220c"
server_key_iv: "53abfc3f406c81bf863fd4c1a895a4e9"
server_key_version: "encryptionkey@0"
I20260924 04:08:50.426903 6467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:08:50.440075 6467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:08:50.443506 6467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:08:50.443717 6467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:08:50.450860 6467 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38949
I20260924 04:08:50.451166 6534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38949 every 8 connection(s)
I20260924 04:08:50.451862 6467 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:08:50.454797 6535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:08:50.457966 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 6467
I20260924 04:08:50.458112 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/master-0/wal/instance
I20260924 04:08:50.458405 392 external_mini_cluster.cc:1428] Setting key 9fa3c4bbde729eb1ace522552af20826
I20260924 04:08:50.461026 6535 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd568f5711864cc4a75a0f2fef7236a2: Bootstrap starting.
I20260924 04:08:50.463637 6535 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd568f5711864cc4a75a0f2fef7236a2: Neither blocks nor log segments found. Creating new log.
I20260924 04:08:50.464529 6535 log.cc:824] T 00000000000000000000000000000000 P cd568f5711864cc4a75a0f2fef7236a2: Log is configured to *not* fsync() on all Append() calls
I20260924 04:08:50.466799 6535 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd568f5711864cc4a75a0f2fef7236a2: No bootstrap required, opened a new log
Sep 24 04:08:50 dist-test-slave-00ld krb5kdc[6454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222930, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 04:08:50.470057 6535 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd568f5711864cc4a75a0f2fef7236a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd568f5711864cc4a75a0f2fef7236a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38949 } }
I20260924 04:08:50.470266 6535 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd568f5711864cc4a75a0f2fef7236a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:08:50.470323 6535 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd568f5711864cc4a75a0f2fef7236a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd568f5711864cc4a75a0f2fef7236a2, State: Initialized, Role: FOLLOWER
I20260924 04:08:50.470738 6535 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd568f5711864cc4a75a0f2fef7236a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd568f5711864cc4a75a0f2fef7236a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38949 } }
I20260924 04:08:50.470854 6535 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd568f5711864cc4a75a0f2fef7236a2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:08:50.470937 6535 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd568f5711864cc4a75a0f2fef7236a2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:08:50.471035 6535 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd568f5711864cc4a75a0f2fef7236a2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:08:50.471897 6535 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd568f5711864cc4a75a0f2fef7236a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd568f5711864cc4a75a0f2fef7236a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38949 } }
I20260924 04:08:50.472227 6535 leader_election.cc:302] T 00000000000000000000000000000000 P cd568f5711864cc4a75a0f2fef7236a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd568f5711864cc4a75a0f2fef7236a2; no voters:
I20260924 04:08:50.472576 6535 leader_election.cc:288] T 00000000000000000000000000000000 P cd568f5711864cc4a75a0f2fef7236a2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:08:50.472678 6540 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd568f5711864cc4a75a0f2fef7236a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:08:50.472936 6540 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd568f5711864cc4a75a0f2fef7236a2 [term 1 LEADER]: Becoming Leader. State: Replica: cd568f5711864cc4a75a0f2fef7236a2, State: Running, Role: LEADER
I20260924 04:08:50.473416 6540 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd568f5711864cc4a75a0f2fef7236a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd568f5711864cc4a75a0f2fef7236a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38949 } }
I20260924 04:08:50.473850 6535 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd568f5711864cc4a75a0f2fef7236a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:08:50.474151 6538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:50.459828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53554 (local address 127.0.98.62:38949)
0924 04:08:50.460243 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:50.460258 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:50.460293 (+ 35us) server_negotiation.cc:407] Connection header received
0924 04:08:50.460984 (+ 691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:50.461003 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:50.461414 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:50.461695 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:50.462641 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:50.463631 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:50.464850 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:50.465624 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:50.465750 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:50.468440 (+ 2690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:50.468444 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:50.468457 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:50.468483 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:50.470987 (+ 2504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:50.471427 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:50.471433 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:50.471438 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:50.471555 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:50.471859 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:50.471862 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:50.471864 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:50.472307 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:50.472509 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:50.472895 (+ 386us) server_negotiation.cc:299] Negotiation successful
0924 04:08:50.473197 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":131,"threads_started":1}
I20260924 04:08:50.475030 6541 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd568f5711864cc4a75a0f2fef7236a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cd568f5711864cc4a75a0f2fef7236a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd568f5711864cc4a75a0f2fef7236a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38949 } } }
I20260924 04:08:50.475152 6541 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd568f5711864cc4a75a0f2fef7236a2 [sys.catalog]: This master's current role is: LEADER
I20260924 04:08:50.475512 6542 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd568f5711864cc4a75a0f2fef7236a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd568f5711864cc4a75a0f2fef7236a2. Latest consensus state: current_term: 1 leader_uuid: "cd568f5711864cc4a75a0f2fef7236a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd568f5711864cc4a75a0f2fef7236a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38949 } } }
I20260924 04:08:50.475663 6542 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd568f5711864cc4a75a0f2fef7236a2 [sys.catalog]: This master's current role is: LEADER
I20260924 04:08:50.476818 6550 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:08:50.479409 6550 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:08:50.484721 6550 catalog_manager.cc:1347] Generated new cluster ID: 66060a64b3b6445ca22111ebb637e30d
I20260924 04:08:50.484794 6550 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:08:50.511294 6550 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:08:50.512262 6550 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:08:50.523919 6550 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cd568f5711864cc4a75a0f2fef7236a2: Generated new TSK 0
I20260924 04:08:50.524788 6550 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260924 04:08:50.555815 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38949
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:08:50.669355 6561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:08:50.669637 6561 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:08:50.669682 6561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:08:50.673893 6561 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:08:50.673961 6561 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:08:50.674080 6561 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:08:50.679737 6561 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38949
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6561
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:08:50.680882 6561 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:08:50.682001 6561 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:08:50.691982 6566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:08:50.691969 6569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:08:50.691972 6567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:08:50.692353 6561 server_base.cc:1034] running on GCE node
I20260924 04:08:50.692840 6561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:08:50.693545 6561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:08:50.694738 6561 hybrid_clock.cc:648] HybridClock initialized: now 1790222930694714 us; error 49 us; skew 500 ppm
Sep 24 04:08:50 dist-test-slave-00ld krb5kdc[6454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222930, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:08:50.698458 6561 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260924 04:08:50.699350 6561 webserver.cc:460] Webserver started at http://127.0.98.1:37605/ using document root <none> and password file <none>
I20260924 04:08:50.699937 6561 fs_manager.cc:359] Metadata directory not provided
I20260924 04:08:50.700031 6561 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:08:50.700268 6561 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:08:50.702365 6561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/ts-0/data/instance:
uuid: "dc6019f5f5474832b90b9150dcc4b0b3"
format_stamp: "Formatted at 2026-09-24 04:08:50 on dist-test-slave-00ld"
server_key: "f7fde6affeee1b71ac96701b9e2831db"
server_key_iv: "65865c46eefa004488380d500cfed5b1"
server_key_version: "encryptionkey@0"
I20260924 04:08:50.702915 6561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/ts-0/wal/instance:
uuid: "dc6019f5f5474832b90b9150dcc4b0b3"
format_stamp: "Formatted at 2026-09-24 04:08:50 on dist-test-slave-00ld"
server_key: "f7fde6affeee1b71ac96701b9e2831db"
server_key_iv: "65865c46eefa004488380d500cfed5b1"
server_key_version: "encryptionkey@0"
I20260924 04:08:50.706691 6561 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260924 04:08:50.709053 6576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:50.710284 6561 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:08:50.710444 6561 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/ts-0/wal
uuid: "dc6019f5f5474832b90b9150dcc4b0b3"
format_stamp: "Formatted at 2026-09-24 04:08:50 on dist-test-slave-00ld"
server_key: "f7fde6affeee1b71ac96701b9e2831db"
server_key_iv: "65865c46eefa004488380d500cfed5b1"
server_key_version: "encryptionkey@0"
I20260924 04:08:50.710562 6561 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:08:50.727113 6561 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:08:50.730399 6561 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:08:50.730631 6561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:08:50.731310 6561 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:08:50.732161 6561 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:08:50.732234 6561 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:50.732314 6561 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:08:50.732366 6561 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:50.741333 6561 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44685
I20260924 04:08:50.741357 6689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44685 every 8 connection(s)
I20260924 04:08:50.742224 6561 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 04:08:50 dist-test-slave-00ld krb5kdc[6454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222930, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 04:08:50.752563 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 6561
I20260924 04:08:50.752740 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:08:50.753044 392 external_mini_cluster.cc:1428] Setting key ddd7cc85d4c4315b86bc5a31b4021bf1
I20260924 04:08:50.756624 6538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:50.744140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42651 (local address 127.0.98.62:38949)
0924 04:08:50.744274 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:50.744278 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:50.745174 (+ 896us) server_negotiation.cc:407] Connection header received
0924 04:08:50.746559 (+ 1385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:50.746562 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:50.746617 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:50.746726 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:50.747883 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:50.748439 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:50.749653 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:50.749884 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:50.749997 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:50.752437 (+ 2440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:50.752440 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:50.752442 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:50.752466 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:50.754198 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:50.754886 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:50.754890 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:50.754893 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:50.755007 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:50.755395 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:50.755401 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:50.755406 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:50.755674 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:50.755785 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:50.756295 (+ 510us) server_negotiation.cc:299] Negotiation successful
0924 04:08:50.756451 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 04:08:50.757921 6692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:50.744532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38949 (local address 127.0.98.1:42651)
0924 04:08:50.745024 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:50.745067 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:50.746332 (+ 1265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:50.746906 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:50.746913 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:50.747265 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:08:50.747734 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:50.747747 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:50.748563 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:50.748566 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:50.749527 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:50.749536 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:50.750035 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:50.750039 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:50.750210 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:50.750783 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:08:50.750798 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:08:50.752216 (+ 1418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:08:50.754409 (+ 2193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:50.754417 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:50.754435 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:50.754774 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:50.755136 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:50.755139 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:50.755140 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:50.755290 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:50.755784 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:08:50.755789 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:08:50.756054 (+ 265us) client_negotiation.cc:769] Sending connection context
0924 04:08:50.756222 (+ 168us) client_negotiation.cc:240] Negotiation successful
0924 04:08:50.756658 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":175,"threads_started":1}
I20260924 04:08:50.759348 6690 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38949
I20260924 04:08:50.759622 6690 heartbeater.cc:464] Registering TS with master...
I20260924 04:08:50.760146 6690 heartbeater.cc:511] Master 127.0.98.62:38949 requested a full tablet report, sending...
I20260924 04:08:50.761855 6499 ts_manager.cc:194] Registered new tserver with Master: dc6019f5f5474832b90b9150dcc4b0b3 (127.0.98.1:44685)
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 04:08:50.763084 6499 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:42651
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260924 04:08:50.783954 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38949
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:08:50.895375 6695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:08:50.895689 6695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:08:50.895773 6695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:08:50.899830 6695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:08:50.899919 6695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:08:50.900034 6695 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:08:50.905283 6695 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38949
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6695
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:08:50.906348 6695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:08:50.907366 6695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:08:50.917238 6703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:08:50.917238 6701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:08:50.917275 6700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:08:50.917416 6695 server_base.cc:1034] running on GCE node
I20260924 04:08:50.918134 6695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:08:50.918767 6695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:08:50.919934 6695 hybrid_clock.cc:648] HybridClock initialized: now 1790222930919912 us; error 27 us; skew 500 ppm
Sep 24 04:08:50 dist-test-slave-00ld krb5kdc[6454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222930, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:08:50.922629 6695 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260924 04:08:50.923513 6695 webserver.cc:460] Webserver started at http://127.0.98.2:36659/ using document root <none> and password file <none>
I20260924 04:08:50.924103 6695 fs_manager.cc:359] Metadata directory not provided
I20260924 04:08:50.924192 6695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:08:50.924422 6695 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:08:50.926538 6695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/ts-1/data/instance:
uuid: "67def1cdfdb048899741175fab806933"
format_stamp: "Formatted at 2026-09-24 04:08:50 on dist-test-slave-00ld"
server_key: "4b115070d325d52350e48672c96df302"
server_key_iv: "91b70d9549107afe937d72f8cbe726a4"
server_key_version: "encryptionkey@0"
I20260924 04:08:50.927126 6695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/ts-1/wal/instance:
uuid: "67def1cdfdb048899741175fab806933"
format_stamp: "Formatted at 2026-09-24 04:08:50 on dist-test-slave-00ld"
server_key: "4b115070d325d52350e48672c96df302"
server_key_iv: "91b70d9549107afe937d72f8cbe726a4"
server_key_version: "encryptionkey@0"
I20260924 04:08:50.931088 6695 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260924 04:08:50.933601 6710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:50.934767 6695 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:08:50.934906 6695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/ts-1/wal
uuid: "67def1cdfdb048899741175fab806933"
format_stamp: "Formatted at 2026-09-24 04:08:50 on dist-test-slave-00ld"
server_key: "4b115070d325d52350e48672c96df302"
server_key_iv: "91b70d9549107afe937d72f8cbe726a4"
server_key_version: "encryptionkey@0"
I20260924 04:08:50.935027 6695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:08:50.954258 6695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:08:50.957675 6695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:08:50.957887 6695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:08:50.958504 6695 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:08:50.959326 6695 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:08:50.959398 6695 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:50.959466 6695 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:08:50.959510 6695 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:50.968637 6695 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37085
I20260924 04:08:50.968705 6823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37085 every 8 connection(s)
I20260924 04:08:50.969547 6695 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:08:50.970135 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 6695
I20260924 04:08:50.970278 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:08:50.970564 392 external_mini_cluster.cc:1428] Setting key 613b7a5af90fff097aceac58e347d928
Sep 24 04:08:50 dist-test-slave-00ld krb5kdc[6454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222930, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
I20260924 04:08:50.985603 6538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:50.971441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45347 (local address 127.0.98.62:38949)
0924 04:08:50.971576 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:50.971580 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:50.972644 (+ 1064us) server_negotiation.cc:407] Connection header received
0924 04:08:50.974165 (+ 1521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:50.974170 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:50.974234 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:50.974323 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:50.975698 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:50.976449 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:50.978388 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:50.978608 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:50.978688 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:50.981755 (+ 3067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:50.981758 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:50.981760 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:50.981783 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:50.983214 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:50.983939 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:50.983942 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:50.983943 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:50.984017 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:50.984368 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:50.984371 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:50.984372 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:50.984616 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:50.984776 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:50.985313 (+ 537us) server_negotiation.cc:299] Negotiation successful
0924 04:08:50.985437 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 04:08:50.986729 6826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:50.971968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38949 (local address 127.0.98.2:45347)
0924 04:08:50.972476 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:50.972529 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:50.973875 (+ 1346us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:50.974517 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:50.974524 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:50.974976 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:08:50.975553 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:50.975574 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:50.976579 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:50.976585 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:50.978251 (+ 1666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:50.978265 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:50.978771 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:50.978778 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:50.979023 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:50.979636 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:08:50.979657 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:08:50.981550 (+ 1893us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:08:50.983341 (+ 1791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:50.983347 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:50.983486 (+ 139us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:50.983837 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:50.984095 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:50.984098 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:50.984099 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:50.984252 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:50.984775 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:08:50.984783 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:08:50.985053 (+ 270us) client_negotiation.cc:769] Sending connection context
0924 04:08:50.985247 (+ 194us) client_negotiation.cc:240] Negotiation successful
0924 04:08:50.985602 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":161,"threads_started":1}
I20260924 04:08:50.988003 6824 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38949
I20260924 04:08:50.988247 6824 heartbeater.cc:464] Registering TS with master...
I20260924 04:08:50.988860 6824 heartbeater.cc:511] Master 127.0.98.62:38949 requested a full tablet report, sending...
I20260924 04:08:50.989935 6499 ts_manager.cc:194] Registered new tserver with Master: 67def1cdfdb048899741175fab806933 (127.0.98.2:37085)
I20260924 04:08:50.990509 6499 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:45347
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260924 04:08:51.001941 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38949
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:08:51.113072 6829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:08:51.113415 6829 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:08:51.113483 6829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:08:51.117715 6829 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:08:51.117782 6829 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:08:51.117842 6829 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:08:51.123102 6829 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38949
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6829
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:08:51.124164 6829 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:08:51.125203 6829 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:08:51.134934 6834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:08:51.134930 6837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:08:51.134927 6835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:08:51.135037 6829 server_base.cc:1034] running on GCE node
I20260924 04:08:51.135645 6829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:08:51.136318 6829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:08:51.137501 6829 hybrid_clock.cc:648] HybridClock initialized: now 1790222931137482 us; error 30 us; skew 500 ppm
Sep 24 04:08:51 dist-test-slave-00ld krb5kdc[6454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222931, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:08:51.140307 6829 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260924 04:08:51.141222 6829 webserver.cc:460] Webserver started at http://127.0.98.3:41897/ using document root <none> and password file <none>
I20260924 04:08:51.141746 6829 fs_manager.cc:359] Metadata directory not provided
I20260924 04:08:51.141803 6829 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:08:51.141968 6829 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:08:51.143963 6829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/ts-2/data/instance:
uuid: "a5a609a5eb2a40cfaf3f6616900bd505"
format_stamp: "Formatted at 2026-09-24 04:08:51 on dist-test-slave-00ld"
server_key: "e7fb8776e1131fbf84c2d80d0dec85cd"
server_key_iv: "f3d3c2ae77ad6ddd6d2ddd061eabb61f"
server_key_version: "encryptionkey@0"
I20260924 04:08:51.144493 6829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a5a609a5eb2a40cfaf3f6616900bd505"
format_stamp: "Formatted at 2026-09-24 04:08:51 on dist-test-slave-00ld"
server_key: "e7fb8776e1131fbf84c2d80d0dec85cd"
server_key_iv: "f3d3c2ae77ad6ddd6d2ddd061eabb61f"
server_key_version: "encryptionkey@0"
I20260924 04:08:51.148205 6829 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 04:08:51.150735 6844 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:51.152043 6829 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 04:08:51.152149 6829 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/ts-2/wal
uuid: "a5a609a5eb2a40cfaf3f6616900bd505"
format_stamp: "Formatted at 2026-09-24 04:08:51 on dist-test-slave-00ld"
server_key: "e7fb8776e1131fbf84c2d80d0dec85cd"
server_key_iv: "f3d3c2ae77ad6ddd6d2ddd061eabb61f"
server_key_version: "encryptionkey@0"
I20260924 04:08:51.152240 6829 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:08:51.168027 6829 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:08:51.171175 6829 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:08:51.171331 6829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:08:51.171865 6829 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:08:51.172632 6829 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:08:51.172729 6829 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:51.172785 6829 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:08:51.172809 6829 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:51.181608 6829 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33301
I20260924 04:08:51.181638 6957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33301 every 8 connection(s)
I20260924 04:08:51.182520 6829 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:08:51.188045 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 6829
I20260924 04:08:51.188179 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:08:51.188486 392 external_mini_cluster.cc:1428] Setting key cdd1ad5ccb393595aee8f22727c6afe7
Sep 24 04:08:51 dist-test-slave-00ld krb5kdc[6454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222931, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 04:08:51.196653 6538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:51.184387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42923 (local address 127.0.98.62:38949)
0924 04:08:51.184598 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:51.184601 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:51.185411 (+ 810us) server_negotiation.cc:407] Connection header received
0924 04:08:51.186763 (+ 1352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:51.186766 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:51.186821 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:51.186903 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:51.188274 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:51.189007 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:51.190518 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:51.190832 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:51.190917 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:51.193331 (+ 2414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:51.193335 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:51.193337 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:51.193360 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:51.194875 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:51.195368 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:51.195371 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:51.195372 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:51.195443 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:51.195721 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:51.195723 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:51.195724 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:51.195884 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:51.195956 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:51.196413 (+ 457us) server_negotiation.cc:299] Negotiation successful
0924 04:08:51.196527 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
I20260924 04:08:51.197811 6960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:51.184825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38949 (local address 127.0.98.3:42923)
0924 04:08:51.185251 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:51.185293 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:51.186542 (+ 1249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:51.187060 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:51.187066 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:51.187463 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:08:51.188064 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:51.188086 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:51.189133 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:51.189138 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:51.190385 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:51.190394 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:51.190970 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:51.190973 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:51.191108 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:51.191678 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:08:51.191693 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:08:51.193138 (+ 1445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:08:51.194998 (+ 1860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:51.195003 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:51.195015 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:51.195265 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:51.195521 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:51.195523 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:51.195524 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:51.195646 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:51.195970 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:08:51.195975 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:08:51.196192 (+ 217us) client_negotiation.cc:769] Sending connection context
0924 04:08:51.196343 (+ 151us) client_negotiation.cc:240] Negotiation successful
0924 04:08:51.196581 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":114,"threads_started":1}
I20260924 04:08:51.199251 6958 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38949
I20260924 04:08:51.199476 6958 heartbeater.cc:464] Registering TS with master...
I20260924 04:08:51.199941 6958 heartbeater.cc:511] Master 127.0.98.62:38949 requested a full tablet report, sending...
I20260924 04:08:51.200997 6499 ts_manager.cc:194] Registered new tserver with Master: a5a609a5eb2a40cfaf3f6616900bd505 (127.0.98.3:33301)
I20260924 04:08:51.201646 6499 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42923
I20260924 04:08:51.202296 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260924 04:08:51.228884 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:43705
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38949,127.0.98.61:43705
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:08:51.342998 6963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:08:51.343413 6963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:08:51.343533 6963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:08:51.349108 6963 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:08:51.349200 6963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:08:51.349241 6963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:08:51.349274 6963 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:08:51.349310 6963 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:08:51.354944 6963 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38949,127.0.98.61:43705
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:43705
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6963
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
W20260924 04:08:51.355208 6963 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:38949,127.0.98.61:43705'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260924 04:08:51.356216 6963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:08:51.357556 6963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:08:51.366925 6968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:08:51.367131 6963 server_base.cc:1034] running on GCE node
W20260924 04:08:51.366959 6969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:08:51.366940 6971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:08:51.368667 6963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:08:51.369872 6963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:08:51.371059 6963 hybrid_clock.cc:648] HybridClock initialized: now 1790222931371034 us; error 47 us; skew 500 ppm
Sep 24 04:08:51 dist-test-slave-00ld krb5kdc[6454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222931, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:08:51.374150 6963 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260924 04:08:51.375069 6963 webserver.cc:460] Webserver started at http://127.0.98.61:44683/ using document root <none> and password file <none>
I20260924 04:08:51.375687 6963 fs_manager.cc:359] Metadata directory not provided
I20260924 04:08:51.375751 6963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:08:51.375970 6963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:08:51.378072 6963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/master-1/data/instance:
uuid: "3c01afc87f8b4560b973c517eea256a2"
format_stamp: "Formatted at 2026-09-24 04:08:51 on dist-test-slave-00ld"
server_key: "f75d04891337896f0515dfef33d1c999"
server_key_iv: "b61264c4d4fa1562966fb4be0329242d"
server_key_version: "encryptionkey@0"
I20260924 04:08:51.378624 6963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/master-1/wal/instance:
uuid: "3c01afc87f8b4560b973c517eea256a2"
format_stamp: "Formatted at 2026-09-24 04:08:51 on dist-test-slave-00ld"
server_key: "f75d04891337896f0515dfef33d1c999"
server_key_iv: "b61264c4d4fa1562966fb4be0329242d"
server_key_version: "encryptionkey@0"
I20260924 04:08:51.383939 6963 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260924 04:08:51.386916 6978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:51.388180 6963 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:08:51.388319 6963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/master-1/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/master-1/wal
uuid: "3c01afc87f8b4560b973c517eea256a2"
format_stamp: "Formatted at 2026-09-24 04:08:51 on dist-test-slave-00ld"
server_key: "f75d04891337896f0515dfef33d1c999"
server_key_iv: "b61264c4d4fa1562966fb4be0329242d"
server_key_version: "encryptionkey@0"
I20260924 04:08:51.388437 6963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:08:51.409734 6963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:08:51.413097 6963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:08:51.413300 6963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:08:51.420559 6963 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:43705
I20260924 04:08:51.420886 7030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:43705 every 8 connection(s)
I20260924 04:08:51.421645 6963 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/master-1/data/info.pb
I20260924 04:08:51.424410 7031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:08:51.425200 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 6963
I20260924 04:08:51.429168 7031 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38949 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 04:08:51 dist-test-slave-00ld krb5kdc[6454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222930, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 24 04:08:51 dist-test-slave-00ld krb5kdc[6454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222931, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 04:08:51.439859 7034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:51.426725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42176 (local address 127.0.98.61:43705)
0924 04:08:51.427174 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:51.427188 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:51.427224 (+ 36us) server_negotiation.cc:407] Connection header received
0924 04:08:51.427895 (+ 671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:51.427914 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:51.428233 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:51.428569 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:51.429432 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:51.430587 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:51.431613 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:51.432071 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:51.432278 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:51.435041 (+ 2763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:51.435046 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:51.435055 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:51.435080 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:51.436939 (+ 1859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:51.437401 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:51.437405 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:51.437409 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:51.437505 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:51.437785 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:51.437790 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:51.437792 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:51.438152 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:51.438371 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:51.438711 (+ 340us) server_negotiation.cc:299] Negotiation successful
0924 04:08:51.438990 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":305,"thread_start_us":149,"threads_started":1}
I20260924 04:08:51.440428 6538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:51.430845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53566 (local address 127.0.98.62:38949)
0924 04:08:51.430994 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:51.430998 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:51.431340 (+ 342us) server_negotiation.cc:407] Connection header received
0924 04:08:51.431542 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:51.431546 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:51.431607 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:51.431651 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:51.432373 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:51.433006 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:51.434058 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:51.434268 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:51.434345 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:51.437073 (+ 2728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:51.437077 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:51.437079 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:51.437101 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:51.438535 (+ 1434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:51.439044 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:51.439047 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:51.439049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:51.439130 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:51.439412 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:51.439415 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:51.439416 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:51.439674 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:51.439782 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:51.440165 (+ 383us) server_negotiation.cc:299] Negotiation successful
0924 04:08:51.440272 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 04:08:51.440469 7035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:51.430824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38949 (local address 127.0.0.1:53566)
0924 04:08:51.431219 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:51.431266 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:51.431434 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:51.431791 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:51.431795 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:51.432107 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:08:51.432250 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:51.432256 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:51.433143 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:51.433149 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:51.433919 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:51.433931 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:51.434423 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:51.434428 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:51.434549 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:51.435072 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:08:51.435084 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:08:51.436932 (+ 1848us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:08:51.438648 (+ 1716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:51.438655 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:51.438661 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:51.438914 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:51.439224 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:51.439229 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:51.439231 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:51.439316 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:51.439778 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:08:51.439784 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:08:51.439933 (+ 149us) client_negotiation.cc:769] Sending connection context
0924 04:08:51.440096 (+ 163us) client_negotiation.cc:240] Negotiation successful
0924 04:08:51.440272 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"mutex_wait_us":1,"thread_start_us":170,"threads_started":1}
I20260924 04:08:51.442056 7031 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43705 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 04:08:51 dist-test-slave-00ld krb5kdc[6454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222931, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260924 04:08:51.449393 7035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:51.442507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43705 (local address 127.0.0.1:42180)
0924 04:08:51.442636 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:51.442649 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:51.442727 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:51.442877 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:51.442879 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:51.443009 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:08:51.443424 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:51.443430 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:51.444099 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:51.444102 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:51.444600 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:51.444607 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:51.445052 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:51.445054 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:51.445124 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:51.445620 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:08:51.445631 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:08:51.446803 (+ 1172us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:08:51.448171 (+ 1368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:51.448173 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:51.448174 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:51.448348 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:51.448571 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:51.448572 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:51.448574 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:51.448642 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:51.449012 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:08:51.449014 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:08:51.449085 (+ 71us) client_negotiation.cc:769] Sending connection context
0924 04:08:51.449163 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 04:08:51.449254 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260924 04:08:51.449421 7034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:51.442583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42180 (local address 127.0.98.61:43705)
0924 04:08:51.442658 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:51.442661 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:51.442695 (+ 34us) server_negotiation.cc:407] Connection header received
0924 04:08:51.442780 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:51.442782 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:51.442812 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:51.442872 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:51.443527 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:51.444010 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:51.444743 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:51.444960 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:51.445038 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:51.446917 (+ 1879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:51.446920 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:51.446922 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:51.446941 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:51.448077 (+ 1136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:51.448428 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:51.448430 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:51.448431 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:51.448495 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:51.448777 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:51.448779 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:51.448780 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:51.448933 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:51.449005 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:51.449215 (+ 210us) server_negotiation.cc:299] Negotiation successful
0924 04:08:51.449300 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 04:08:51.452716 7031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c01afc87f8b4560b973c517eea256a2: Bootstrap starting.
I20260924 04:08:51.455384 7031 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c01afc87f8b4560b973c517eea256a2: Neither blocks nor log segments found. Creating new log.
I20260924 04:08:51.456226 7031 log.cc:824] T 00000000000000000000000000000000 P 3c01afc87f8b4560b973c517eea256a2: Log is configured to *not* fsync() on all Append() calls
I20260924 04:08:51.458419 7031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c01afc87f8b4560b973c517eea256a2: No bootstrap required, opened a new log
I20260924 04:08:51.462421 7031 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c01afc87f8b4560b973c517eea256a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd568f5711864cc4a75a0f2fef7236a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38949 } } peers { permanent_uuid: "3c01afc87f8b4560b973c517eea256a2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43705 } }
I20260924 04:08:51.462679 7031 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3c01afc87f8b4560b973c517eea256a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:08:51.462735 7031 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c01afc87f8b4560b973c517eea256a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c01afc87f8b4560b973c517eea256a2, State: Initialized, Role: FOLLOWER
I20260924 04:08:51.463217 7031 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c01afc87f8b4560b973c517eea256a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd568f5711864cc4a75a0f2fef7236a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38949 } } peers { permanent_uuid: "3c01afc87f8b4560b973c517eea256a2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43705 } }
I20260924 04:08:51.463712 7037 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c01afc87f8b4560b973c517eea256a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd568f5711864cc4a75a0f2fef7236a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38949 } } peers { permanent_uuid: "3c01afc87f8b4560b973c517eea256a2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43705 } } }
I20260924 04:08:51.463860 7037 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c01afc87f8b4560b973c517eea256a2 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:08:51.464573 7031 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c01afc87f8b4560b973c517eea256a2 [sys.catalog]: configured and running, proceeding with master startup.
W20260924 04:08:51.469434 7048 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3c01afc87f8b4560b973c517eea256a2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:08:51.469523 7048 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:08:51.471575 6963 master_runner.cc:428] Detected that this master 3c01afc87f8b4560b973c517eea256a2 is joining an existing cluster
I20260924 04:08:51.471639 6963 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:43705
I20260924 04:08:51.482383 6538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:51.473756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53582 (local address 127.0.98.62:38949)
0924 04:08:51.473873 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:51.473880 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:51.474094 (+ 214us) server_negotiation.cc:407] Connection header received
0924 04:08:51.474293 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:51.474299 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:51.474348 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:51.474462 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:51.475068 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:51.475592 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:51.476548 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:51.476831 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:51.476942 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:51.478110 (+ 1168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:51.478113 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:51.478116 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:51.478139 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:51.479721 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:51.481058 (+ 1337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:51.481061 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:51.481063 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:51.481146 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:51.481464 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:51.481469 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:51.481474 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:51.481657 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:51.481745 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:51.482131 (+ 386us) server_negotiation.cc:299] Negotiation successful
0924 04:08:51.482233 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 04:08:51.482409 7034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:51.473741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42188 (local address 127.0.98.61:43705)
0924 04:08:51.473861 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:51.473865 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:51.474304 (+ 439us) server_negotiation.cc:407] Connection header received
0924 04:08:51.474481 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:51.474484 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:51.474521 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:51.474612 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:51.475167 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:51.475896 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:51.477022 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:51.477384 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:51.477469 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:51.478707 (+ 1238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:51.478711 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:51.478713 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:51.478734 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:51.480472 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:51.481051 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:51.481056 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:51.481057 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:51.481140 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:51.481427 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:51.481430 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:51.481432 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:51.481630 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:51.481733 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:51.482020 (+ 287us) server_negotiation.cc:299] Negotiation successful
0924 04:08:51.482245 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":142,"server-negotiator.queue_time_us":39}
I20260924 04:08:51.482474 7054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:51.473681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38949 (local address 127.0.0.1:53582)
0924 04:08:51.474011 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:51.474028 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:51.474182 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:51.474603 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:51.474606 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:51.474862 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:08:51.474955 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:51.474960 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:51.475701 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:51.475707 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:51.476425 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:51.476434 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:51.476950 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:51.476953 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:51.477069 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:51.477635 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:08:51.477646 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:08:51.477974 (+ 328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:08:51.480590 (+ 2616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:51.480592 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:51.480594 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:51.480942 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:51.481258 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:51.481261 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:51.481263 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:51.481346 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:51.481794 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:08:51.481797 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:08:51.481864 (+ 67us) client_negotiation.cc:769] Sending connection context
0924 04:08:51.481974 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 04:08:51.482119 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"mutex_wait_us":20,"thread_start_us":98,"threads_started":1}
I20260924 04:08:51.482786 7055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:51.473951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43705 (local address 127.0.0.1:42188)
0924 04:08:51.474229 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:51.474244 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:51.474351 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:51.474647 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:51.474650 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:51.474900 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:08:51.475049 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:51.475056 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:51.476024 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:51.476027 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:51.476894 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:51.476904 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:51.477614 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:51.477617 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:51.477719 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:51.478177 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:08:51.478192 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:08:51.478577 (+ 385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:08:51.480585 (+ 2008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:51.480587 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:51.480589 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:51.480904 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:51.481230 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:51.481234 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:51.481236 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:51.481314 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:51.481740 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:08:51.481743 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:08:51.481835 (+ 92us) client_negotiation.cc:769] Sending connection context
0924 04:08:51.481980 (+ 145us) client_negotiation.cc:240] Negotiation successful
0924 04:08:51.482522 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"mutex_wait_us":72,"thread_start_us":93,"threads_started":1}
Sep 24 04:08:51 dist-test-slave-00ld krb5kdc[6454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222930, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260924 04:08:51.497255 7034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:51.487448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42204 (local address 127.0.98.61:43705)
0924 04:08:51.487705 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:51.487709 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:51.487853 (+ 144us) server_negotiation.cc:407] Connection header received
0924 04:08:51.488033 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:51.488037 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:51.488086 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:51.488169 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:51.488807 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:51.489554 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:51.490398 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:51.490729 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:51.490833 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:51.493517 (+ 2684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:51.493521 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:51.493524 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:51.493548 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:51.495187 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:51.495766 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:51.495769 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:51.495771 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:51.495853 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:51.496182 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:51.496185 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:51.496187 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:51.496399 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:51.496496 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:51.496960 (+ 464us) server_negotiation.cc:299] Negotiation successful
0924 04:08:51.497079 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188}
I20260924 04:08:51.497716 7056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:51.487359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43705 (local address 127.0.0.1:42204)
0924 04:08:51.487727 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:51.487775 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:51.487902 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:51.488204 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:51.488208 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:51.488550 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:08:51.488649 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:51.488655 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:51.489664 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:51.489667 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:51.490287 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:51.490295 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:51.490839 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:51.490842 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:51.490925 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:51.491463 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:08:51.491474 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:08:51.493270 (+ 1796us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:08:51.495301 (+ 2031us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:51.495313 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:51.495321 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:51.495660 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:51.495966 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:51.495972 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:51.495976 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:51.496085 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:51.496519 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:08:51.496529 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:08:51.496710 (+ 181us) client_negotiation.cc:769] Sending connection context
0924 04:08:51.496880 (+ 170us) client_negotiation.cc:240] Negotiation successful
0924 04:08:51.497113 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":104,"threads_started":1}
I20260924 04:08:51.499563 6499 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:43705: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "3c01afc87f8b4560b973c517eea256a2"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 43705
}
attrs {
promote: true
}
}
dest_uuid: "cd568f5711864cc4a75a0f2fef7236a2"
cas_config_opid_index: -1
I20260924 04:08:51.500316 6499 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd568f5711864cc4a75a0f2fef7236a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "cd568f5711864cc4a75a0f2fef7236a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38949 } } peers { permanent_uuid: "3c01afc87f8b4560b973c517eea256a2" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43705 } attrs { promote: true } }
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260924 04:08:51.502332 7059 raft_consensus.cc:2932] T 00000000000000000000000000000000 P cd568f5711864cc4a75a0f2fef7236a2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 3c01afc87f8b4560b973c517eea256a2 (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "cd568f5711864cc4a75a0f2fef7236a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38949 } } peers { permanent_uuid: "3c01afc87f8b4560b973c517eea256a2" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43705 } attrs { promote: true } } }
I20260924 04:08:51.502802 7005 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c01afc87f8b4560b973c517eea256a2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:08:51.504864 7059 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:43705
I20260924 04:08:51.504868 7005 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3c01afc87f8b4560b973c517eea256a2 [term 1 FOLLOWER]: Refusing update from remote peer cd568f5711864cc4a75a0f2fef7236a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260924 04:08:51.505267 6963 master.cc:537] Master@127.0.98.61:43705 shutting down...
I20260924 04:08:51.505517 7059 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cd568f5711864cc4a75a0f2fef7236a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c01afc87f8b4560b973c517eea256a2" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43705 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
W20260924 04:08:51.507227 6484 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260924 04:08:51.505339 7058 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd568f5711864cc4a75a0f2fef7236a2 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "cd568f5711864cc4a75a0f2fef7236a2" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "cd568f5711864cc4a75a0f2fef7236a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38949 } } peers { permanent_uuid: "3c01afc87f8b4560b973c517eea256a2" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43705 } attrs { promote: true } } }
I20260924 04:08:51.507689 7058 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd568f5711864cc4a75a0f2fef7236a2 [sys.catalog]: This master's current role is: LEADER
W20260924 04:08:51.508114 6484 consensus_peers.cc:487] T 00000000000000000000000000000000 P cd568f5711864cc4a75a0f2fef7236a2 -> Peer 3c01afc87f8b4560b973c517eea256a2 (127.0.98.61:43705): Couldn't send request to peer 3c01afc87f8b4560b973c517eea256a2. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260924 04:08:51.510288 6963 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 3c01afc87f8b4560b973c517eea256a2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:08:51.510576 6963 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 3c01afc87f8b4560b973c517eea256a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:08:51.510689 6963 tablet_replica.cc:330] T 00000000000000000000000000000000 P 3c01afc87f8b4560b973c517eea256a2: stopping tablet replica
I20260924 04:08:51.515899 6963 master.cc:552] Master@127.0.98.61:43705 shutdown complete.
I20260924 04:08:51.515954 6963 master_runner.cc:305] Clearing existing system tablet
I20260924 04:08:51.516191 6963 fs_manager.cc:359] Metadata directory not provided
I20260924 04:08:51.516252 6963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:08:51.517580 6963 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 04:08:51.518771 7064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:51.519047 6963 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:08:51.519162 6963 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 04:08:51.519194 6963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/master-1/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/master-1/wal
uuid: "3c01afc87f8b4560b973c517eea256a2"
format_stamp: "Formatted at 2026-09-24 04:08:51 on dist-test-slave-00ld"
server_key: "f75d04891337896f0515dfef33d1c999"
server_key_iv: "b61264c4d4fa1562966fb4be0329242d"
server_key_version: "encryptionkey@0"
I20260924 04:08:51.521236 6963 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 3c01afc87f8b4560b973c517eea256a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:08:51.524530 6963 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 3c01afc87f8b4560b973c517eea256a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 04:08:51.524621 6963 log.cc:1196] T 00000000000000000000000000000000 P 3c01afc87f8b4560b973c517eea256a2: Deleting WAL directory at /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260924 04:08:51.524919 6963 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 3c01afc87f8b4560b973c517eea256a2: Deleting consensus metadata
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260924 04:08:51.525086 6963 master_runner.cc:315] Copying system tablet from 127.0.98.62:38949
I20260924 04:08:51.525921 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:34051
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38949,127.0.98.61:43705,127.0.98.60:34051
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:08:51.526216 6963 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 3c01afc87f8b4560b973c517eea256a2: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:38949
I20260924 04:08:51.534952 6538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:51.526866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53588 (local address 127.0.98.62:38949)
0924 04:08:51.527001 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:51.527006 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:51.527218 (+ 212us) server_negotiation.cc:407] Connection header received
0924 04:08:51.527400 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:51.527403 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:51.527454 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:51.527507 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:51.528132 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:51.528930 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:51.530047 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:51.530279 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:51.530368 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:51.531774 (+ 1406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:51.531777 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:51.531779 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:51.531799 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:51.533280 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:51.533777 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:51.533779 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:51.533780 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:51.533855 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:51.534167 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:51.534169 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:51.534171 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:51.534329 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:51.534401 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:51.534732 (+ 331us) server_negotiation.cc:299] Negotiation successful
0924 04:08:51.534822 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 04:08:51.535249 7070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:51.526774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38949 (local address 127.0.0.1:53588)
0924 04:08:51.527129 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:51.527144 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:51.527291 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:51.527643 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:51.527646 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:51.527858 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:08:51.527983 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:51.527989 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:51.529046 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:51.529049 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:51.529885 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:51.529894 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:51.530405 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:51.530409 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:51.530559 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:51.531122 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:08:51.531144 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:08:51.531640 (+ 496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:08:51.533390 (+ 1750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:51.533396 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:51.533399 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:51.533679 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:51.533951 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:51.533957 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:51.533961 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:51.534067 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:51.534424 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:08:51.534430 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:08:51.534558 (+ 128us) client_negotiation.cc:769] Sending connection context
0924 04:08:51.534687 (+ 129us) client_negotiation.cc:240] Negotiation successful
0924 04:08:51.534859 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":90,"threads_started":1}
I20260924 04:08:51.535873 6519 tablet_copy_service.cc:140] P cd568f5711864cc4a75a0f2fef7236a2: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 3c01afc87f8b4560b973c517eea256a2 ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:53588)
I20260924 04:08:51.536098 6519 tablet_copy_service.cc:161] P cd568f5711864cc4a75a0f2fef7236a2: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 3c01afc87f8b4560b973c517eea256a2 at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:53588: session id = 3c01afc87f8b4560b973c517eea256a2-00000000000000000000000000000000
I20260924 04:08:51.538401 6519 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P cd568f5711864cc4a75a0f2fef7236a2: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 04:08:51.540071 6963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 04:08:51.542418 6963 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 3c01afc87f8b4560b973c517eea256a2: tablet copy: Starting download of 0 data blocks...
I20260924 04:08:51.542608 6963 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 3c01afc87f8b4560b973c517eea256a2: tablet copy: Starting download of 1 WAL segments...
I20260924 04:08:51.544371 6963 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 3c01afc87f8b4560b973c517eea256a2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 04:08:51.546275 6519 tablet_copy_service.cc:342] P cd568f5711864cc4a75a0f2fef7236a2: Request end of tablet copy session 3c01afc87f8b4560b973c517eea256a2-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:53588
I20260924 04:08:51.546394 6519 tablet_copy_service.cc:433] P cd568f5711864cc4a75a0f2fef7236a2: ending tablet copy session 3c01afc87f8b4560b973c517eea256a2-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 3c01afc87f8b4560b973c517eea256a2
I20260924 04:08:51.554499 6963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:08:51.556185 7075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:08:51.556221 7072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:08:51.556766 6963 server_base.cc:1034] running on GCE node
W20260924 04:08:51.556926 7073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:08:51.557147 6963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:08:51.557451 6963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:08:51.558589 6963 hybrid_clock.cc:648] HybridClock initialized: now 1790222931558565 us; error 40 us; skew 500 ppm
Sep 24 04:08:51 dist-test-slave-00ld krb5kdc[6454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222931, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:08:51.560221 6963 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260924 04:08:51.560673 6963 webserver.cc:460] Webserver started at http://127.0.98.61:45469/ using document root <none> and password file <none>
I20260924 04:08:51.560914 6963 fs_manager.cc:359] Metadata directory not provided
I20260924 04:08:51.560958 6963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:08:51.562258 6963 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 04:08:51.563653 7082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:51.563928 6963 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 04:08:51.564002 6963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/master-1/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/master-1/wal
uuid: "3c01afc87f8b4560b973c517eea256a2"
format_stamp: "Formatted at 2026-09-24 04:08:51 on dist-test-slave-00ld"
server_key: "f75d04891337896f0515dfef33d1c999"
server_key_iv: "b61264c4d4fa1562966fb4be0329242d"
server_key_version: "encryptionkey@0"
I20260924 04:08:51.564082 6963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:08:51.578151 6963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:08:51.580381 6963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:08:51.584921 6963 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:43705
I20260924 04:08:51.585023 7134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:43705 every 8 connection(s)
I20260924 04:08:51.585460 6963 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/master-1/data/info.pb
I20260924 04:08:51.587100 7135 sys_catalog.cc:263] Verifying existing consensus state
I20260924 04:08:51.588039 7135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c01afc87f8b4560b973c517eea256a2: Bootstrap starting.
I20260924 04:08:51.597702 7135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c01afc87f8b4560b973c517eea256a2: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 04:08:51.598019 7135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c01afc87f8b4560b973c517eea256a2: Bootstrap complete.
I20260924 04:08:51.598546 7135 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c01afc87f8b4560b973c517eea256a2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "cd568f5711864cc4a75a0f2fef7236a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38949 } } peers { permanent_uuid: "3c01afc87f8b4560b973c517eea256a2" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43705 } attrs { promote: true } }
I20260924 04:08:51.598678 7135 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c01afc87f8b4560b973c517eea256a2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3c01afc87f8b4560b973c517eea256a2, State: Initialized, Role: LEARNER
I20260924 04:08:51.598773 7135 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c01afc87f8b4560b973c517eea256a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "cd568f5711864cc4a75a0f2fef7236a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38949 } } peers { permanent_uuid: "3c01afc87f8b4560b973c517eea256a2" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43705 } attrs { promote: true } }
I20260924 04:08:51.599092 7138 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c01afc87f8b4560b973c517eea256a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "cd568f5711864cc4a75a0f2fef7236a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38949 } } peers { permanent_uuid: "3c01afc87f8b4560b973c517eea256a2" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43705 } attrs { promote: true } } }
I20260924 04:08:51.599195 7135 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c01afc87f8b4560b973c517eea256a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:08:51.599291 7138 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c01afc87f8b4560b973c517eea256a2 [sys.catalog]: This master's current role is: LEARNER
I20260924 04:08:51.602035 7149 catalog_manager.cc:1259] Loaded cluster ID: 66060a64b3b6445ca22111ebb637e30d
I20260924 04:08:51.602087 7149 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3c01afc87f8b4560b973c517eea256a2: loading cluster ID for follower catalog manager: success
I20260924 04:08:51.604082 7149 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3c01afc87f8b4560b973c517eea256a2: acquiring CA information for follower catalog manager: success
I20260924 04:08:51.605440 7149 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3c01afc87f8b4560b973c517eea256a2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260924 04:08:51.656936 7071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:08:51.657215 7071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:08:51.657313 7071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:08:51.661360 7071 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:08:51.661437 7071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:08:51.661458 7071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:08:51.661475 7071 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:08:51.661491 7071 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:08:51.667806 7071 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38949,127.0.98.61:43705,127.0.98.60:34051
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:34051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7071
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:08:51.669009 7071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:08:51.670035 7071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:08:51.678537 7154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:08:51.678638 7155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:08:51.678586 7157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:08:51.678789 7071 server_base.cc:1034] running on GCE node
I20260924 04:08:51.679452 7071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:08:51.680480 7071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:08:51.681636 7071 hybrid_clock.cc:648] HybridClock initialized: now 1790222931681615 us; error 34 us; skew 500 ppm
Sep 24 04:08:51 dist-test-slave-00ld krb5kdc[6454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222931, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:08:51.684342 7071 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260924 04:08:51.685290 7071 webserver.cc:460] Webserver started at http://127.0.98.60:43273/ using document root <none> and password file <none>
I20260924 04:08:51.685845 7071 fs_manager.cc:359] Metadata directory not provided
I20260924 04:08:51.685940 7071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:08:51.686169 7071 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:08:51.688154 7071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/master-2/data/instance:
uuid: "90d32bb2c3d041d78f61550aa83a8a15"
format_stamp: "Formatted at 2026-09-24 04:08:51 on dist-test-slave-00ld"
server_key: "76a5b9c2687f5e72e854d1b2e361660d"
server_key_iv: "85620f2ca472d21f0835014a0d6f3005"
server_key_version: "encryptionkey@0"
I20260924 04:08:51.688736 7071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/master-2/wal/instance:
uuid: "90d32bb2c3d041d78f61550aa83a8a15"
format_stamp: "Formatted at 2026-09-24 04:08:51 on dist-test-slave-00ld"
server_key: "76a5b9c2687f5e72e854d1b2e361660d"
server_key_iv: "85620f2ca472d21f0835014a0d6f3005"
server_key_version: "encryptionkey@0"
I20260924 04:08:51.692255 7071 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260924 04:08:51.694713 7164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:51.695967 7071 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:08:51.696110 7071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/master-2/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/master-2/wal
uuid: "90d32bb2c3d041d78f61550aa83a8a15"
format_stamp: "Formatted at 2026-09-24 04:08:51 on dist-test-slave-00ld"
server_key: "76a5b9c2687f5e72e854d1b2e361660d"
server_key_iv: "85620f2ca472d21f0835014a0d6f3005"
server_key_version: "encryptionkey@0"
I20260924 04:08:51.696226 7071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:08:51.718429 7071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:08:51.721813 7071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:08:51.722024 7071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:08:51.729733 7071 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:34051
I20260924 04:08:51.730037 7216 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:34051 every 8 connection(s)
I20260924 04:08:51.730777 7071 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/master-2/data/info.pb
I20260924 04:08:51.733786 7217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:08:51.735415 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 7071
I20260924 04:08:51.738411 7217 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38949 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 04:08:51 dist-test-slave-00ld krb5kdc[6454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222930, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 24 04:08:51 dist-test-slave-00ld krb5kdc[6454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222931, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 04:08:51.750319 6538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:51.740449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53604 (local address 127.0.98.62:38949)
0924 04:08:51.740571 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:51.740575 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:51.740754 (+ 179us) server_negotiation.cc:407] Connection header received
0924 04:08:51.740895 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:51.740898 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:51.740947 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:51.741036 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:51.741755 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:51.742367 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:51.743614 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:51.743947 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:51.744005 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:51.746676 (+ 2671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:51.746679 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:51.746681 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:51.746704 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:51.748454 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:51.748966 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:51.748970 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:51.748972 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:51.749050 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:51.749384 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:51.749387 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:51.749388 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:51.749561 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:51.749662 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:51.750023 (+ 361us) server_negotiation.cc:299] Negotiation successful
0924 04:08:51.750137 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 04:08:51.751127 7219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:51.736993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49600 (local address 127.0.98.60:34051)
0924 04:08:51.737459 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:51.737474 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:51.737509 (+ 35us) server_negotiation.cc:407] Connection header received
0924 04:08:51.738112 (+ 603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:51.738127 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:51.738513 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:51.738839 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:51.739771 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:51.741128 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:51.742302 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:51.743064 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:51.743211 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:51.746599 (+ 3388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:51.746606 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:51.746623 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:51.746645 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:51.748350 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:51.748819 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:51.748829 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:51.748830 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:51.748933 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:51.749208 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:51.749212 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:51.749214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:51.749611 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:51.749827 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:51.750029 (+ 202us) server_negotiation.cc:299] Negotiation successful
0924 04:08:51.750299 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":61,"server-negotiator.queue_time_us":321,"thread_start_us":176,"threads_started":1}
I20260924 04:08:51.751127 7220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:51.740256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38949 (local address 127.0.0.1:53604)
0924 04:08:51.740623 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:51.740669 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:51.740803 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:51.741103 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:51.741108 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:51.741513 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:08:51.741629 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:51.741635 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:51.742520 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:51.742523 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:51.743458 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:51.743465 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:51.744153 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:51.744155 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:51.744225 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:51.744737 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:08:51.744763 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:08:51.746512 (+ 1749us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:08:51.748558 (+ 2046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:51.748567 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:51.748576 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:51.748869 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:51.749148 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:51.749151 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:51.749153 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:51.749266 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:51.749671 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:08:51.749680 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:08:51.749810 (+ 130us) client_negotiation.cc:769] Sending connection context
0924 04:08:51.749967 (+ 157us) client_negotiation.cc:240] Negotiation successful
0924 04:08:51.750392 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"mutex_wait_us":115,"thread_start_us":110,"threads_started":1}
I20260924 04:08:51.752990 7217 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43705 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 04:08:51 dist-test-slave-00ld krb5kdc[6454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222931, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260924 04:08:51.760774 7220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:51.753440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43705 (local address 127.0.0.1:42218)
0924 04:08:51.753550 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:51.753565 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:51.753633 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:51.753958 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:51.753960 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:51.754081 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:08:51.754162 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:51.754166 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:51.755014 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:51.755016 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:51.755524 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:51.755531 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:51.756019 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:51.756022 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:51.756093 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:51.756539 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:08:51.756550 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:08:51.757801 (+ 1251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:08:51.759293 (+ 1492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:51.759298 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:51.759300 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:51.759523 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:51.759809 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:51.759811 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:51.759814 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:51.759892 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:51.760248 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:08:51.760251 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:08:51.760348 (+ 97us) client_negotiation.cc:769] Sending connection context
0924 04:08:51.760452 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 04:08:51.760572 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260924 04:08:51.760802 7221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:51.753541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42218 (local address 127.0.98.61:43705)
0924 04:08:51.753768 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:51.753772 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:51.753783 (+ 11us) server_negotiation.cc:407] Connection header received
0924 04:08:51.753816 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:51.753818 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:51.753862 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:51.753932 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:51.754269 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:51.754927 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:51.755647 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:51.755924 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:51.756004 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:51.757939 (+ 1935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:51.757942 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:51.757943 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:51.757964 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:51.759184 (+ 1220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:51.759630 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:51.759634 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:51.759637 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:51.759713 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:51.759980 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:51.759985 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:51.759987 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:51.760155 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:51.760259 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:51.760500 (+ 241us) server_negotiation.cc:299] Negotiation successful
0924 04:08:51.760616 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":91,"threads_started":1}
I20260924 04:08:51.761529 7217 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34051 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:08:51.765769 6690 heartbeater.cc:503] Master 127.0.98.62:38949 was elected leader, sending a full tablet report...
Sep 24 04:08:51 dist-test-slave-00ld krb5kdc[6454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222931, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260924 04:08:51.769393 7220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:51.761958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34051 (local address 127.0.0.1:49604)
0924 04:08:51.762069 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:51.762084 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:51.762153 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:51.762333 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:51.762335 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:51.762467 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:08:51.762531 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:51.762535 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:51.763581 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:51.763583 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:51.764067 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:51.764075 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:51.764581 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:51.764583 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:51.764661 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:51.765198 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:08:51.765211 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:08:51.766646 (+ 1435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:08:51.768184 (+ 1538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:51.768187 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:51.768188 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:51.768368 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:51.768598 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:51.768600 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:51.768602 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:51.768669 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:51.769038 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:08:51.769040 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:08:51.769110 (+ 70us) client_negotiation.cc:769] Sending connection context
0924 04:08:51.769187 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 04:08:51.769277 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":93}
I20260924 04:08:51.769469 7219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:51.762036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49604 (local address 127.0.98.60:34051)
0924 04:08:51.762137 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:51.762141 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:51.762153 (+ 12us) server_negotiation.cc:407] Connection header received
0924 04:08:51.762214 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:51.762216 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:51.762248 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:51.762305 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:51.762622 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:51.763475 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:51.764164 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:51.764486 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:51.764639 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:51.766768 (+ 2129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:51.766771 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:51.766774 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:51.766795 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:51.768092 (+ 1297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:51.768451 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:51.768454 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:51.768455 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:51.768524 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:51.768803 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:51.768806 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:51.768807 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:51.768961 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:51.769032 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:51.769243 (+ 211us) server_negotiation.cc:299] Negotiation successful
0924 04:08:51.769335 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":41}
I20260924 04:08:51.773074 7217 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90d32bb2c3d041d78f61550aa83a8a15: Bootstrap starting.
I20260924 04:08:51.775517 7217 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 90d32bb2c3d041d78f61550aa83a8a15: Neither blocks nor log segments found. Creating new log.
I20260924 04:08:51.776170 7217 log.cc:824] T 00000000000000000000000000000000 P 90d32bb2c3d041d78f61550aa83a8a15: Log is configured to *not* fsync() on all Append() calls
I20260924 04:08:51.778026 7217 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90d32bb2c3d041d78f61550aa83a8a15: No bootstrap required, opened a new log
I20260924 04:08:51.780915 7217 raft_consensus.cc:348] T 00000000000000000000000000000000 P 90d32bb2c3d041d78f61550aa83a8a15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd568f5711864cc4a75a0f2fef7236a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38949 } } peers { permanent_uuid: "3c01afc87f8b4560b973c517eea256a2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43705 } } peers { permanent_uuid: "90d32bb2c3d041d78f61550aa83a8a15" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34051 } }
I20260924 04:08:51.781108 7217 raft_consensus.cc:374] T 00000000000000000000000000000000 P 90d32bb2c3d041d78f61550aa83a8a15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:08:51.781140 7217 raft_consensus.cc:729] T 00000000000000000000000000000000 P 90d32bb2c3d041d78f61550aa83a8a15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d32bb2c3d041d78f61550aa83a8a15, State: Initialized, Role: FOLLOWER
I20260924 04:08:51.781517 7217 consensus_queue.cc:260] T 00000000000000000000000000000000 P 90d32bb2c3d041d78f61550aa83a8a15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd568f5711864cc4a75a0f2fef7236a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38949 } } peers { permanent_uuid: "3c01afc87f8b4560b973c517eea256a2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43705 } } peers { permanent_uuid: "90d32bb2c3d041d78f61550aa83a8a15" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34051 } }
I20260924 04:08:51.781915 7223 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90d32bb2c3d041d78f61550aa83a8a15 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd568f5711864cc4a75a0f2fef7236a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38949 } } peers { permanent_uuid: "3c01afc87f8b4560b973c517eea256a2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43705 } } peers { permanent_uuid: "90d32bb2c3d041d78f61550aa83a8a15" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34051 } } }
I20260924 04:08:51.782052 7223 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90d32bb2c3d041d78f61550aa83a8a15 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:08:51.782608 7217 sys_catalog.cc:564] T 00000000000000000000000000000000 P 90d32bb2c3d041d78f61550aa83a8a15 [sys.catalog]: configured and running, proceeding with master startup.
W20260924 04:08:51.787319 7234 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 90d32bb2c3d041d78f61550aa83a8a15: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:08:51.787421 7234 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:08:51.790833 7071 master_runner.cc:428] Detected that this master 90d32bb2c3d041d78f61550aa83a8a15 is joining an existing cluster
I20260924 04:08:51.790900 7071 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:34051
I20260924 04:08:51.804308 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 6561
I20260924 04:08:51.810863 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:44685
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:38949
--webserver_port=37605
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:38949,127.0.98.61:43705
--tserver_master_addrs=127.0.98.62:38949,127.0.98.61:43705,127.0.98.60:34051 with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:08:51.821624 7221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:51.794008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42220 (local address 127.0.98.61:43705)
0924 04:08:51.794125 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:51.794131 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:51.794145 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:08:51.794184 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:51.794187 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:51.794241 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:51.794315 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:51.795209 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:51.795995 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:51.797405 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:51.797744 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:51.797809 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:51.817883 (+ 20074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:51.817888 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:51.817891 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:51.817938 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:51.819732 (+ 1794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:51.820229 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:51.820232 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:51.820234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:51.820314 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:51.820598 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:51.820601 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:51.820603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:51.820882 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:51.820953 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:51.821311 (+ 358us) server_negotiation.cc:299] Negotiation successful
0924 04:08:51.821453 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 04:08:51.822185 7240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:51.793253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43705 (local address 127.0.0.1:42220)
0924 04:08:51.793565 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:51.793581 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:51.793674 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:51.794460 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:51.794464 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:51.794909 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:08:51.795085 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:51.795092 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:51.796134 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:51.796137 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:51.797262 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:51.797272 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:51.797985 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:51.797989 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:51.798172 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:51.798724 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:08:51.798738 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:08:51.817659 (+ 18921us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:08:51.819872 (+ 2213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:51.819875 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:51.819877 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:51.820109 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:51.820422 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:51.820425 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:51.820427 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:51.820507 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:51.821023 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:08:51.821026 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:08:51.821143 (+ 117us) client_negotiation.cc:769] Sending connection context
0924 04:08:51.821845 (+ 702us) client_negotiation.cc:240] Negotiation successful
0924 04:08:51.822001 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"mutex_wait_us":244,"spinlock_wait_cycles":5248,"thread_start_us":105,"threads_started":1}
I20260924 04:08:51.823788 7219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:51.794183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49618 (local address 127.0.98.60:34051)
0924 04:08:51.794279 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:51.794284 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:51.794295 (+ 11us) server_negotiation.cc:407] Connection header received
0924 04:08:51.794326 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:51.794329 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:51.794369 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:51.794438 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:51.795158 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:51.795956 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:51.797594 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:51.798376 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:51.798463 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:51.818822 (+ 20359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:51.818826 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:51.818829 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:51.818853 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:51.821195 (+ 2342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:51.822681 (+ 1486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:51.822684 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:51.822685 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:51.822748 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:51.823001 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:51.823004 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:51.823005 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:51.823178 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:51.823240 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:51.823531 (+ 291us) server_negotiation.cc:299] Negotiation successful
0924 04:08:51.823634 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":41}
I20260924 04:08:51.824223 7242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:51.793685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34051 (local address 127.0.0.1:49618)
0924 04:08:51.793931 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:51.793943 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:51.794031 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:51.794536 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:51.794539 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:51.794883 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:08:51.795038 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:51.795047 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:51.796075 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:51.796078 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:51.797472 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:51.797482 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:51.798628 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:51.798632 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:51.798737 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:51.799137 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:08:51.799151 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:08:51.818364 (+ 19213us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:08:51.822352 (+ 3988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:51.822355 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:51.822356 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:51.822585 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:51.822849 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:51.822852 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:51.822854 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:51.822922 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:51.823294 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:08:51.823297 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:08:51.823368 (+ 71us) client_negotiation.cc:769] Sending connection context
0924 04:08:51.823951 (+ 583us) client_negotiation.cc:240] Negotiation successful
0924 04:08:51.824068 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":4,"thread_start_us":73,"threads_started":1}
I20260924 04:08:51.826354 7241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:51.793253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38949 (local address 127.0.0.1:53612)
0924 04:08:51.793653 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:51.793667 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:51.793764 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:51.794476 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:51.794479 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:51.794924 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:08:51.795059 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:51.795069 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:51.796134 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:51.796137 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:51.797262 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:51.797272 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:51.797985 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:51.797989 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:51.798702 (+ 713us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:51.817436 (+ 18734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:08:51.817459 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:08:51.817948 (+ 489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:08:51.824590 (+ 6642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:51.824596 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:51.824600 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:51.824925 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:51.825269 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:51.825272 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:51.825274 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:51.825371 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:51.825883 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:08:51.825886 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:08:51.825971 (+ 85us) client_negotiation.cc:769] Sending connection context
0924 04:08:51.826062 (+ 91us) client_negotiation.cc:240] Negotiation successful
0924 04:08:51.826184 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"mutex_wait_us":161,"thread_start_us":111,"threads_started":1}
I20260924 04:08:51.826508 6538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:51.794031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53612 (local address 127.0.98.62:38949)
0924 04:08:51.794206 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:51.794210 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:51.794223 (+ 13us) server_negotiation.cc:407] Connection header received
0924 04:08:51.794260 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:51.794266 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:51.794314 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:51.794377 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:51.795202 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:51.796001 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:51.797397 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:51.797728 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:51.797805 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:51.818137 (+ 20332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:51.818144 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:51.818150 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:51.818473 (+ 323us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:51.822227 (+ 3754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:51.825063 (+ 2836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:51.825068 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:51.825072 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:51.825168 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:51.825492 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:51.825498 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:51.825501 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:51.825773 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:51.825886 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:51.826167 (+ 281us) server_negotiation.cc:299] Negotiation successful
0924 04:08:51.826307 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
Sep 24 04:08:51 dist-test-slave-00ld krb5kdc[6454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222930, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260924 04:08:51.839344 7056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:51.830312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34051 (local address 127.0.0.1:49620)
0924 04:08:51.830422 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:51.830436 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:51.830526 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:51.830812 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:51.830815 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:51.831006 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:08:51.831112 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:51.831119 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:51.832159 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:51.832162 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:51.833050 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:51.833060 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:51.833692 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:51.833695 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:51.833808 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:51.834264 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:08:51.834278 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:08:51.835812 (+ 1534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:08:51.837804 (+ 1992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:51.837808 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:51.837812 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:51.838059 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:51.838349 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:51.838353 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:51.838355 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:51.838440 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:51.838853 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:08:51.838856 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:08:51.838948 (+ 92us) client_negotiation.cc:769] Sending connection context
0924 04:08:51.839042 (+ 94us) client_negotiation.cc:240] Negotiation successful
0924 04:08:51.839174 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 04:08:51.840049 7219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:51.830468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49620 (local address 127.0.98.60:34051)
0924 04:08:51.830571 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:51.830575 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:51.830587 (+ 12us) server_negotiation.cc:407] Connection header received
0924 04:08:51.830650 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:51.830653 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:51.830700 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:51.830778 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:51.831251 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:51.832010 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:51.833180 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:51.833499 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:51.833585 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:51.835945 (+ 2360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:51.835949 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:51.835952 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:51.835977 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:51.837686 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:51.838157 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:51.838160 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:51.838162 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:51.838240 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:51.838535 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:51.838538 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:51.838540 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:51.838719 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:51.838809 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:51.839800 (+ 991us) server_negotiation.cc:299] Negotiation successful
0924 04:08:51.839906 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 04:08:51.841094 6499 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:34051: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "90d32bb2c3d041d78f61550aa83a8a15"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 34051
}
attrs {
promote: true
}
}
dest_uuid: "cd568f5711864cc4a75a0f2fef7236a2"
cas_config_opid_index: 5
I20260924 04:08:51.841692 6499 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd568f5711864cc4a75a0f2fef7236a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "cd568f5711864cc4a75a0f2fef7236a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38949 } } peers { permanent_uuid: "3c01afc87f8b4560b973c517eea256a2" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43705 } attrs { promote: true } } peers { permanent_uuid: "90d32bb2c3d041d78f61550aa83a8a15" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 34051 } attrs { promote: true } }
I20260924 04:08:51.842715 7058 raft_consensus.cc:2932] T 00000000000000000000000000000000 P cd568f5711864cc4a75a0f2fef7236a2 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 90d32bb2c3d041d78f61550aa83a8a15 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "cd568f5711864cc4a75a0f2fef7236a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38949 } } peers { permanent_uuid: "3c01afc87f8b4560b973c517eea256a2" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43705 } attrs { promote: true } } peers { permanent_uuid: "90d32bb2c3d041d78f61550aa83a8a15" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 34051 } attrs { promote: true } } }
I20260924 04:08:51.844270 7058 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:34051
I20260924 04:08:51.844558 7059 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd568f5711864cc4a75a0f2fef7236a2 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "cd568f5711864cc4a75a0f2fef7236a2" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "cd568f5711864cc4a75a0f2fef7236a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38949 } } peers { permanent_uuid: "3c01afc87f8b4560b973c517eea256a2" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43705 } attrs { promote: true } } peers { permanent_uuid: "90d32bb2c3d041d78f61550aa83a8a15" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 34051 } attrs { promote: true } } }
I20260924 04:08:51.844657 7059 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd568f5711864cc4a75a0f2fef7236a2 [sys.catalog]: This master's current role is: LEADER
I20260924 04:08:51.845623 7071 master.cc:537] Master@127.0.98.60:34051 shutting down...
I20260924 04:08:51.845588 7191 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 90d32bb2c3d041d78f61550aa83a8a15 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:08:51.846982 7191 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 90d32bb2c3d041d78f61550aa83a8a15 [term 1 FOLLOWER]: Refusing update from remote peer cd568f5711864cc4a75a0f2fef7236a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260924 04:08:51.847757 7059 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cd568f5711864cc4a75a0f2fef7236a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d32bb2c3d041d78f61550aa83a8a15" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 34051 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 04:08:51.849520 6484 consensus_peers.cc:487] T 00000000000000000000000000000000 P cd568f5711864cc4a75a0f2fef7236a2 -> Peer 90d32bb2c3d041d78f61550aa83a8a15 (127.0.98.60:34051): Couldn't send request to peer 90d32bb2c3d041d78f61550aa83a8a15. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260924 04:08:51.851586 7056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:51.844671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43705 (local address 127.0.0.1:42228)
0924 04:08:51.844966 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:51.844981 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:51.845052 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:51.846456 (+ 1404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:51.846459 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:51.846476 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:08:51.846557 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:51.846563 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:51.847585 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:51.847588 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:51.849848 (+ 2260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:51.849858 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:51.851177 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:51.851180 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:51.851269 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:51.851315 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 04:08:51.851388 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 04:08:51.851463 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183}
I20260924 04:08:51.851989 7221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:51.844873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42228 (local address 127.0.98.61:43705)
0924 04:08:51.846093 (+ 1220us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:51.846097 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:51.846112 (+ 15us) server_negotiation.cc:407] Connection header received
0924 04:08:51.846164 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:51.846166 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:51.846223 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:51.846314 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:08:51.846682 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:51.847454 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:51.849995 (+ 2541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:51.850799 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:51.850914 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:51.851111 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:51.851788 (+ 677us) server_negotiation.cc:299] Negotiation successful
0924 04:08:51.851865 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1135}
I20260924 04:08:51.852425 7109 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3c01afc87f8b4560b973c517eea256a2 [term 1 LEARNER]: Refusing update from remote peer cd568f5711864cc4a75a0f2fef7236a2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260924 04:08:51.852916 7059 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cd568f5711864cc4a75a0f2fef7236a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c01afc87f8b4560b973c517eea256a2" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43705 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260924 04:08:51.853222 7071 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 90d32bb2c3d041d78f61550aa83a8a15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:08:51.853485 7071 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 90d32bb2c3d041d78f61550aa83a8a15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:08:51.853572 7071 tablet_replica.cc:330] T 00000000000000000000000000000000 P 90d32bb2c3d041d78f61550aa83a8a15: stopping tablet replica
I20260924 04:08:51.854784 7109 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3c01afc87f8b4560b973c517eea256a2 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 90d32bb2c3d041d78f61550aa83a8a15 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "cd568f5711864cc4a75a0f2fef7236a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38949 } } peers { permanent_uuid: "3c01afc87f8b4560b973c517eea256a2" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43705 } attrs { promote: true } } peers { permanent_uuid: "90d32bb2c3d041d78f61550aa83a8a15" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 34051 } attrs { promote: true } } }
I20260924 04:08:51.856741 7138 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c01afc87f8b4560b973c517eea256a2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cd568f5711864cc4a75a0f2fef7236a2" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "cd568f5711864cc4a75a0f2fef7236a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38949 } } peers { permanent_uuid: "3c01afc87f8b4560b973c517eea256a2" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43705 } attrs { promote: true } } peers { permanent_uuid: "90d32bb2c3d041d78f61550aa83a8a15" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 34051 } attrs { promote: true } } }
I20260924 04:08:51.856868 7138 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c01afc87f8b4560b973c517eea256a2 [sys.catalog]: This master's current role is: LEARNER
I20260924 04:08:51.856788 7245 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c01afc87f8b4560b973c517eea256a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd568f5711864cc4a75a0f2fef7236a2. Latest consensus state: current_term: 1 leader_uuid: "cd568f5711864cc4a75a0f2fef7236a2" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "cd568f5711864cc4a75a0f2fef7236a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38949 } } peers { permanent_uuid: "3c01afc87f8b4560b973c517eea256a2" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43705 } attrs { promote: true } } peers { permanent_uuid: "90d32bb2c3d041d78f61550aa83a8a15" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 34051 } attrs { promote: true } } }
I20260924 04:08:51.857067 7245 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c01afc87f8b4560b973c517eea256a2 [sys.catalog]: This master's current role is: LEARNER
I20260924 04:08:51.858457 7058 raft_consensus.cc:1050] T 00000000000000000000000000000000 P cd568f5711864cc4a75a0f2fef7236a2: attempting to promote NON_VOTER 3c01afc87f8b4560b973c517eea256a2 to VOTER
I20260924 04:08:51.858899 7058 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd568f5711864cc4a75a0f2fef7236a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cd568f5711864cc4a75a0f2fef7236a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38949 } } peers { permanent_uuid: "3c01afc87f8b4560b973c517eea256a2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43705 } attrs { promote: false } } peers { permanent_uuid: "90d32bb2c3d041d78f61550aa83a8a15" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 34051 } attrs { promote: true } }
I20260924 04:08:51.860204 7109 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3c01afc87f8b4560b973c517eea256a2 [term 1 LEARNER]: Refusing update from remote peer cd568f5711864cc4a75a0f2fef7236a2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260924 04:08:51.860844 7059 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cd568f5711864cc4a75a0f2fef7236a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c01afc87f8b4560b973c517eea256a2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43705 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260924 04:08:51.861022 6484 consensus_peers.cc:487] T 00000000000000000000000000000000 P cd568f5711864cc4a75a0f2fef7236a2 -> Peer 90d32bb2c3d041d78f61550aa83a8a15 (127.0.98.60:34051): Couldn't send request to peer 90d32bb2c3d041d78f61550aa83a8a15. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260924 04:08:51.862583 7059 raft_consensus.cc:2932] T 00000000000000000000000000000000 P cd568f5711864cc4a75a0f2fef7236a2 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 3c01afc87f8b4560b973c517eea256a2 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cd568f5711864cc4a75a0f2fef7236a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38949 } } peers { permanent_uuid: "3c01afc87f8b4560b973c517eea256a2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43705 } attrs { promote: false } } peers { permanent_uuid: "90d32bb2c3d041d78f61550aa83a8a15" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 34051 } attrs { promote: true } } }
I20260924 04:08:51.864064 7059 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd568f5711864cc4a75a0f2fef7236a2 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "cd568f5711864cc4a75a0f2fef7236a2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cd568f5711864cc4a75a0f2fef7236a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38949 } } peers { permanent_uuid: "3c01afc87f8b4560b973c517eea256a2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43705 } attrs { promote: false } } peers { permanent_uuid: "90d32bb2c3d041d78f61550aa83a8a15" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 34051 } attrs { promote: true } } }
I20260924 04:08:51.864161 7059 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd568f5711864cc4a75a0f2fef7236a2 [sys.catalog]: This master's current role is: LEADER
I20260924 04:08:51.864275 7059 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd568f5711864cc4a75a0f2fef7236a2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cd568f5711864cc4a75a0f2fef7236a2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cd568f5711864cc4a75a0f2fef7236a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38949 } } peers { permanent_uuid: "3c01afc87f8b4560b973c517eea256a2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43705 } attrs { promote: false } } peers { permanent_uuid: "90d32bb2c3d041d78f61550aa83a8a15" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 34051 } attrs { promote: true } } }
I20260924 04:08:51.864360 7059 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd568f5711864cc4a75a0f2fef7236a2 [sys.catalog]: This master's current role is: LEADER
I20260924 04:08:51.865200 7109 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3c01afc87f8b4560b973c517eea256a2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 3c01afc87f8b4560b973c517eea256a2 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cd568f5711864cc4a75a0f2fef7236a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38949 } } peers { permanent_uuid: "3c01afc87f8b4560b973c517eea256a2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43705 } attrs { promote: false } } peers { permanent_uuid: "90d32bb2c3d041d78f61550aa83a8a15" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 34051 } attrs { promote: true } } }
I20260924 04:08:51.865540 7071 master.cc:552] Master@127.0.98.60:34051 shutdown complete.
I20260924 04:08:51.865583 7071 master_runner.cc:305] Clearing existing system tablet
I20260924 04:08:51.865811 7071 fs_manager.cc:359] Metadata directory not provided
I20260924 04:08:51.865854 7071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:08:51.867014 7071 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 04:08:51.867295 7245 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c01afc87f8b4560b973c517eea256a2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cd568f5711864cc4a75a0f2fef7236a2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cd568f5711864cc4a75a0f2fef7236a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38949 } } peers { permanent_uuid: "3c01afc87f8b4560b973c517eea256a2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43705 } attrs { promote: false } } peers { permanent_uuid: "90d32bb2c3d041d78f61550aa83a8a15" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 34051 } attrs { promote: true } } }
I20260924 04:08:51.867481 7245 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c01afc87f8b4560b973c517eea256a2 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:08:51.867823 7248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:51.868070 7071 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:08:51.868208 7071 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 04:08:51.868249 7071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/master-2/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/master-2/wal
uuid: "90d32bb2c3d041d78f61550aa83a8a15"
format_stamp: "Formatted at 2026-09-24 04:08:51 on dist-test-slave-00ld"
server_key: "76a5b9c2687f5e72e854d1b2e361660d"
server_key_iv: "85620f2ca472d21f0835014a0d6f3005"
server_key_version: "encryptionkey@0"
I20260924 04:08:51.870355 7071 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 90d32bb2c3d041d78f61550aa83a8a15: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:08:51.872869 7071 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 90d32bb2c3d041d78f61550aa83a8a15: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 04:08:51.872967 7071 log.cc:1196] T 00000000000000000000000000000000 P 90d32bb2c3d041d78f61550aa83a8a15: Deleting WAL directory at /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260924 04:08:51.873198 7071 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 90d32bb2c3d041d78f61550aa83a8a15: Deleting consensus metadata
I20260924 04:08:51.873567 7071 master_runner.cc:315] Copying system tablet from 127.0.98.62:38949
I20260924 04:08:51.874723 7071 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 90d32bb2c3d041d78f61550aa83a8a15: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:38949
I20260924 04:08:51.884055 7254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:51.875230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38949 (local address 127.0.0.1:53616)
0924 04:08:51.875501 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:51.875515 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:51.875626 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:51.875909 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:51.875912 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:51.876128 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:08:51.876254 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:51.876261 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:51.877389 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:51.877393 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:51.878291 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:51.878301 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:51.878850 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:51.878854 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:51.878942 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:51.879532 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:08:51.879547 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:08:51.879975 (+ 428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:08:51.882128 (+ 2153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:51.882131 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:51.882133 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:51.882440 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:51.882854 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:51.882856 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:51.882859 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:51.882957 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:51.883473 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:08:51.883476 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:08:51.883582 (+ 106us) client_negotiation.cc:769] Sending connection context
0924 04:08:51.883692 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 04:08:51.883839 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":81,"threads_started":1}
I20260924 04:08:51.884086 6538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:51.875314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53616 (local address 127.0.98.62:38949)
0924 04:08:51.875424 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:51.875429 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:51.875597 (+ 168us) server_negotiation.cc:407] Connection header received
0924 04:08:51.875705 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:51.875708 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:51.875757 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:51.875845 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:51.876379 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:51.877235 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:51.878420 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:51.878728 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:51.878842 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:51.880117 (+ 1275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:51.880121 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:51.880124 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:51.880150 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:51.881993 (+ 1843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:51.882644 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:51.882647 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:51.882649 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:51.882738 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:51.883052 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:51.883056 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:51.883058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:51.883370 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:51.883472 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:51.883811 (+ 339us) server_negotiation.cc:299] Negotiation successful
0924 04:08:51.883930 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 04:08:51.885134 6519 tablet_copy_service.cc:140] P cd568f5711864cc4a75a0f2fef7236a2: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 90d32bb2c3d041d78f61550aa83a8a15 ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:53616)
I20260924 04:08:51.885232 6519 tablet_copy_service.cc:161] P cd568f5711864cc4a75a0f2fef7236a2: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 90d32bb2c3d041d78f61550aa83a8a15 at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:53616: session id = 90d32bb2c3d041d78f61550aa83a8a15-00000000000000000000000000000000
I20260924 04:08:51.886294 6519 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P cd568f5711864cc4a75a0f2fef7236a2: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 04:08:51.887596 7071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 04:08:51.889971 7071 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 90d32bb2c3d041d78f61550aa83a8a15: tablet copy: Starting download of 0 data blocks...
I20260924 04:08:51.890158 7071 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 90d32bb2c3d041d78f61550aa83a8a15: tablet copy: Starting download of 1 WAL segments...
I20260924 04:08:51.891695 7071 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 90d32bb2c3d041d78f61550aa83a8a15: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 04:08:51.893471 6519 tablet_copy_service.cc:342] P cd568f5711864cc4a75a0f2fef7236a2: Request end of tablet copy session 90d32bb2c3d041d78f61550aa83a8a15-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:53616
I20260924 04:08:51.893550 6519 tablet_copy_service.cc:433] P cd568f5711864cc4a75a0f2fef7236a2: ending tablet copy session 90d32bb2c3d041d78f61550aa83a8a15-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 90d32bb2c3d041d78f61550aa83a8a15
I20260924 04:08:51.901317 7071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:08:51.902954 7255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:08:51.905818 7258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:08:51.905900 7071 server_base.cc:1034] running on GCE node
W20260924 04:08:51.909185 7256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:08:51.909443 7071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:08:51.909773 7071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:08:51.910933 7071 hybrid_clock.cc:648] HybridClock initialized: now 1790222931910915 us; error 41 us; skew 500 ppm
Sep 24 04:08:51 dist-test-slave-00ld krb5kdc[6454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222931, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:08:51.912611 7071 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260924 04:08:51.913131 7071 webserver.cc:460] Webserver started at http://127.0.98.60:39711/ using document root <none> and password file <none>
I20260924 04:08:51.913323 7071 fs_manager.cc:359] Metadata directory not provided
I20260924 04:08:51.913365 7071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:08:51.914722 7071 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 04:08:51.915758 7265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:51.916030 7071 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 04:08:51.916105 7071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/master-2/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/master-2/wal
uuid: "90d32bb2c3d041d78f61550aa83a8a15"
format_stamp: "Formatted at 2026-09-24 04:08:51 on dist-test-slave-00ld"
server_key: "76a5b9c2687f5e72e854d1b2e361660d"
server_key_iv: "85620f2ca472d21f0835014a0d6f3005"
server_key_version: "encryptionkey@0"
I20260924 04:08:51.916203 7071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:08:51.930801 7071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:08:51.933117 7071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:08:51.938071 7071 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:34051
I20260924 04:08:51.938107 7317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:34051 every 8 connection(s)
I20260924 04:08:51.938743 7071 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/master-2/data/info.pb
I20260924 04:08:51.940799 7318 sys_catalog.cc:263] Verifying existing consensus state
I20260924 04:08:51.941936 7318 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90d32bb2c3d041d78f61550aa83a8a15: Bootstrap starting.
I20260924 04:08:51.952980 7318 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90d32bb2c3d041d78f61550aa83a8a15: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 04:08:51.953446 7318 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90d32bb2c3d041d78f61550aa83a8a15: Bootstrap complete.
I20260924 04:08:51.954141 7318 raft_consensus.cc:348] T 00000000000000000000000000000000 P 90d32bb2c3d041d78f61550aa83a8a15 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cd568f5711864cc4a75a0f2fef7236a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38949 } } peers { permanent_uuid: "3c01afc87f8b4560b973c517eea256a2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43705 } attrs { promote: false } } peers { permanent_uuid: "90d32bb2c3d041d78f61550aa83a8a15" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 34051 } attrs { promote: true } }
I20260924 04:08:51.954381 7318 raft_consensus.cc:729] T 00000000000000000000000000000000 P 90d32bb2c3d041d78f61550aa83a8a15 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 90d32bb2c3d041d78f61550aa83a8a15, State: Initialized, Role: LEARNER
I20260924 04:08:51.954530 7318 consensus_queue.cc:260] T 00000000000000000000000000000000 P 90d32bb2c3d041d78f61550aa83a8a15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cd568f5711864cc4a75a0f2fef7236a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38949 } } peers { permanent_uuid: "3c01afc87f8b4560b973c517eea256a2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43705 } attrs { promote: false } } peers { permanent_uuid: "90d32bb2c3d041d78f61550aa83a8a15" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 34051 } attrs { promote: true } }
I20260924 04:08:51.954972 7324 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90d32bb2c3d041d78f61550aa83a8a15 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cd568f5711864cc4a75a0f2fef7236a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38949 } } peers { permanent_uuid: "3c01afc87f8b4560b973c517eea256a2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43705 } attrs { promote: false } } peers { permanent_uuid: "90d32bb2c3d041d78f61550aa83a8a15" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 34051 } attrs { promote: true } } }
I20260924 04:08:51.955145 7324 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90d32bb2c3d041d78f61550aa83a8a15 [sys.catalog]: This master's current role is: LEARNER
I20260924 04:08:51.955241 7318 sys_catalog.cc:564] T 00000000000000000000000000000000 P 90d32bb2c3d041d78f61550aa83a8a15 [sys.catalog]: configured and running, proceeding with master startup.
W20260924 04:08:51.955860 7243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:08:51.956229 7243 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:08:51.956343 7243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260924 04:08:51.959277 7335 catalog_manager.cc:1259] Loaded cluster ID: 66060a64b3b6445ca22111ebb637e30d
I20260924 04:08:51.959343 7335 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 90d32bb2c3d041d78f61550aa83a8a15: loading cluster ID for follower catalog manager: success
I20260924 04:08:51.961483 7335 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 90d32bb2c3d041d78f61550aa83a8a15: acquiring CA information for follower catalog manager: success
W20260924 04:08:51.962510 7243 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:08:51.962579 7243 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:08:51.962661 7243 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:08:51.962838 7335 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 90d32bb2c3d041d78f61550aa83a8a15: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 04:08:51.968291 7243 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:44685
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=37605
--tserver_master_addrs=127.0.98.62:38949,127.0.98.61:43705,127.0.98.60:34051
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7243
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:08:51.969403 7243 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:08:51.970376 7243 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:08:51.980296 7338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:08:51.980258 7337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:08:51.980348 7243 server_base.cc:1034] running on GCE node
W20260924 04:08:51.980258 7340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:08:51.981014 7243 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:08:51.981693 7243 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:08:51.982903 7243 hybrid_clock.cc:648] HybridClock initialized: now 1790222931982873 us; error 41 us; skew 500 ppm
Sep 24 04:08:51 dist-test-slave-00ld krb5kdc[6454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222931, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:08:51.985832 7243 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260924 04:08:51.986757 7243 webserver.cc:460] Webserver started at http://127.0.98.1:37605/ using document root <none> and password file <none>
I20260924 04:08:51.987332 7243 fs_manager.cc:359] Metadata directory not provided
I20260924 04:08:51.987460 7243 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:08:51.991343 7243 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:08:51.992535 6824 heartbeater.cc:503] Master 127.0.98.62:38949 was elected leader, sending a full tablet report...
I20260924 04:08:51.993599 7347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:51.994985 7243 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 04:08:51.995086 7243 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/ts-0/wal
uuid: "dc6019f5f5474832b90b9150dcc4b0b3"
format_stamp: "Formatted at 2026-09-24 04:08:50 on dist-test-slave-00ld"
server_key: "f7fde6affeee1b71ac96701b9e2831db"
server_key_iv: "65865c46eefa004488380d500cfed5b1"
server_key_version: "encryptionkey@0"
I20260924 04:08:51.995594 7243 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:08:52.020215 7243 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:08:52.023324 7243 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:08:52.023483 7243 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:08:52.024019 7243 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:08:52.024852 7243 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:08:52.024897 7243 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:52.024937 7243 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:08:52.024952 7243 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:52.034745 7243 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44685
I20260924 04:08:52.034770 7460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44685 every 8 connection(s)
I20260924 04:08:52.035750 7243 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:08:52.041895 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 7243
I20260924 04:08:52.042114 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 6695
Sep 24 04:08:52 dist-test-slave-00ld krb5kdc[6454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222931, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 24 04:08:52 dist-test-slave-00ld krb5kdc[6454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222931, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 24 04:08:52 dist-test-slave-00ld krb5kdc[6454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222931, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 04:08:52.051191 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:37085
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:38949
--webserver_port=36659
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:38949,127.0.98.61:43705
--tserver_master_addrs=127.0.98.62:38949,127.0.98.61:43705,127.0.98.60:34051 with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:08:52.055122 7221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:52.037673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42139 (local address 127.0.98.61:43705)
0924 04:08:52.037787 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:52.037791 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:52.039172 (+ 1381us) server_negotiation.cc:407] Connection header received
0924 04:08:52.040269 (+ 1097us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:52.040273 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:52.040332 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:52.040408 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:52.042074 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:52.042915 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:52.045000 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:52.045442 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:52.045534 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:52.050122 (+ 4588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:52.050129 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:52.050135 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:52.050169 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:52.052624 (+ 2455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:52.053351 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:52.053355 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:52.053356 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:52.053435 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:52.053770 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:52.053773 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:52.053775 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:52.053994 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:52.054076 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:52.054837 (+ 761us) server_negotiation.cc:299] Negotiation successful
0924 04:08:52.054957 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 04:08:52.055294 6538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:52.037627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39821 (local address 127.0.98.62:38949)
0924 04:08:52.037763 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:52.037767 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:52.039203 (+ 1436us) server_negotiation.cc:407] Connection header received
0924 04:08:52.040392 (+ 1189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:52.040396 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:52.040443 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:52.040538 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:52.042071 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:52.042892 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:52.044981 (+ 2089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:52.045323 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:52.045412 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:52.050423 (+ 5011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:52.050429 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:52.050435 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:52.050469 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:52.052594 (+ 2125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:52.053358 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:52.053363 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:52.053365 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:52.053461 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:52.053792 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:52.053797 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:52.053800 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:52.054049 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:52.054139 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:52.054847 (+ 708us) server_negotiation.cc:299] Negotiation successful
0924 04:08:52.054998 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 04:08:52.056174 7467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:52.039214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34051 (local address 127.0.98.1:35349)
0924 04:08:52.039527 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:52.039540 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:52.040057 (+ 517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:52.040730 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:52.040733 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:52.041158 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:08:52.041903 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:52.041922 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:52.043237 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:52.043240 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:52.044844 (+ 1604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:52.044854 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:52.045582 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:52.045586 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:52.045982 (+ 396us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:52.046956 (+ 974us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:08:52.046979 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:08:52.049911 (+ 2932us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:08:52.052623 (+ 2712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:52.052629 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:52.052648 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:52.053168 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:52.053487 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:52.053490 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:52.053492 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:52.053670 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:52.054131 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:08:52.054138 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:08:52.054560 (+ 422us) client_negotiation.cc:769] Sending connection context
0924 04:08:52.054766 (+ 206us) client_negotiation.cc:240] Negotiation successful
0924 04:08:52.055113 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"mutex_wait_us":535,"spinlock_wait_cycles":69376,"thread_start_us":74,"threads_started":1}
I20260924 04:08:52.056190 7465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:52.038058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43705 (local address 127.0.98.1:42139)
0924 04:08:52.038559 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:52.038609 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:52.040058 (+ 1449us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:52.040595 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:52.040602 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:52.041149 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:08:52.041903 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:52.041924 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:52.043068 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:52.043074 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:52.044863 (+ 1789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:52.044877 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:52.045724 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:52.045730 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:52.046032 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:52.047105 (+ 1073us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:08:52.047125 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:08:52.049913 (+ 2788us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:08:52.052769 (+ 2856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:52.052772 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:52.052774 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:52.053237 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:52.053536 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:52.053539 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:52.053540 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:52.053664 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:52.054131 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:08:52.054138 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:08:52.054554 (+ 416us) client_negotiation.cc:769] Sending connection context
0924 04:08:52.055377 (+ 823us) client_negotiation.cc:240] Negotiation successful
0924 04:08:52.055548 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"mutex_wait_us":540,"spinlock_wait_cycles":24960,"thread_start_us":556,"threads_started":1}
I20260924 04:08:52.056923 7468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:52.039371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35349 (local address 127.0.98.60:34051)
0924 04:08:52.039632 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:52.039671 (+ 39us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:52.039685 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:08:52.040555 (+ 870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:52.040558 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:52.040607 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:52.040719 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:52.042070 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:52.043126 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:52.044980 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:52.045323 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:52.045412 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:52.050113 (+ 4701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:52.050117 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:52.050120 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:52.050150 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:52.052484 (+ 2334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:52.053285 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:52.053291 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:52.053295 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:52.053383 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:52.053769 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:52.053773 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:52.053775 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:52.053994 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:52.054077 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:52.056589 (+ 2512us) server_negotiation.cc:299] Negotiation successful
0924 04:08:52.056759 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":84,"threads_started":1}
I20260924 04:08:52.057526 7466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:52.038058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38949 (local address 127.0.98.1:39821)
0924 04:08:52.038824 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:52.038837 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:52.040057 (+ 1220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:52.040593 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:52.040604 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:52.041151 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:08:52.041903 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:52.041922 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:52.043063 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:52.043066 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:52.044847 (+ 1781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:52.044854 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:52.045582 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:52.045586 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:52.045985 (+ 399us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:52.046965 (+ 980us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:08:52.046979 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:08:52.050259 (+ 3280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:08:52.052756 (+ 2497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:52.052761 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:52.052766 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:52.053236 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:52.053575 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:52.053579 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:52.053581 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:52.053676 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:52.054200 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:08:52.054206 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:08:52.054559 (+ 353us) client_negotiation.cc:769] Sending connection context
0924 04:08:52.056062 (+ 1503us) client_negotiation.cc:240] Negotiation successful
0924 04:08:52.056909 (+ 847us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"mutex_wait_us":645,"spinlock_wait_cycles":70016,"thread_start_us":152,"threads_started":1}
I20260924 04:08:52.057806 7463 heartbeater.cc:347] Connected to a master server at 127.0.98.61:43705
I20260924 04:08:52.057893 7461 heartbeater.cc:347] Connected to a master server at 127.0.98.60:34051
I20260924 04:08:52.058056 7461 heartbeater.cc:464] Registering TS with master...
I20260924 04:08:52.058056 7463 heartbeater.cc:464] Registering TS with master...
I20260924 04:08:52.058511 7462 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38949
I20260924 04:08:52.058599 7462 heartbeater.cc:464] Registering TS with master...
I20260924 04:08:52.058671 7463 heartbeater.cc:511] Master 127.0.98.61:43705 requested a full tablet report, sending...
I20260924 04:08:52.058806 7462 heartbeater.cc:511] Master 127.0.98.62:38949 requested a full tablet report, sending...
I20260924 04:08:52.059186 7461 heartbeater.cc:511] Master 127.0.98.60:34051 requested a full tablet report, sending...
I20260924 04:08:52.059960 6499 ts_manager.cc:194] Re-registered known tserver with Master: dc6019f5f5474832b90b9150dcc4b0b3 (127.0.98.1:44685)
I20260924 04:08:52.060137 7098 ts_manager.cc:194] Registered new tserver with Master: dc6019f5f5474832b90b9150dcc4b0b3 (127.0.98.1:44685)
I20260924 04:08:52.060871 6499 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:39821
I20260924 04:08:52.061093 7274 ts_manager.cc:194] Registered new tserver with Master: dc6019f5f5474832b90b9150dcc4b0b3 (127.0.98.1:44685)
W20260924 04:08:52.163643 7469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:08:52.164000 7469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:08:52.164073 7469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:08:52.168414 7469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:08:52.168481 7469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:08:52.168543 7469 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:08:52.173874 7469 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:37085
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=36659
--tserver_master_addrs=127.0.98.62:38949,127.0.98.61:43705,127.0.98.60:34051
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7469
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:08:52.174988 7469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:08:52.176066 7469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:08:52.185714 7474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:08:52.185714 7477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:08:52.185717 7475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:08:52.186146 7469 server_base.cc:1034] running on GCE node
I20260924 04:08:52.186522 7469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:08:52.187063 7469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:08:52.188246 7469 hybrid_clock.cc:648] HybridClock initialized: now 1790222932188216 us; error 48 us; skew 500 ppm
Sep 24 04:08:52 dist-test-slave-00ld krb5kdc[6454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222932, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:08:52.190933 7469 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260924 04:08:52.191769 7469 webserver.cc:460] Webserver started at http://127.0.98.2:36659/ using document root <none> and password file <none>
I20260924 04:08:52.192267 7469 fs_manager.cc:359] Metadata directory not provided
I20260924 04:08:52.192335 7469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:08:52.195984 7469 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260924 04:08:52.197984 7484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:52.199285 7469 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:08:52.199472 7469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/ts-1/wal
uuid: "67def1cdfdb048899741175fab806933"
format_stamp: "Formatted at 2026-09-24 04:08:50 on dist-test-slave-00ld"
server_key: "4b115070d325d52350e48672c96df302"
server_key_iv: "91b70d9549107afe937d72f8cbe726a4"
server_key_version: "encryptionkey@0"
I20260924 04:08:52.200019 7469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:08:52.203950 6958 heartbeater.cc:503] Master 127.0.98.62:38949 was elected leader, sending a full tablet report...
I20260924 04:08:52.221217 7469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:08:52.224565 7469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:08:52.224799 7469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:08:52.225373 7469 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:08:52.226215 7469 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:08:52.226264 7469 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:52.226305 7469 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:08:52.226320 7469 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:52.235543 7469 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37085
I20260924 04:08:52.235628 7597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37085 every 8 connection(s)
I20260924 04:08:52.236578 7469 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:08:52.238333 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 7469
I20260924 04:08:52.238524 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 6829
I20260924 04:08:52.246424 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:33301
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:38949
--webserver_port=41897
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:38949,127.0.98.61:43705
--tserver_master_addrs=127.0.98.62:38949,127.0.98.61:43705,127.0.98.60:34051 with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 04:08:52 dist-test-slave-00ld krb5kdc[6454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222932, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 24 04:08:52 dist-test-slave-00ld krb5kdc[6454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222932, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 24 04:08:52 dist-test-slave-00ld krb5kdc[6454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222932, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260924 04:08:52.255119 6538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:52.238443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57623 (local address 127.0.98.62:38949)
0924 04:08:52.238613 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:52.238619 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:52.241417 (+ 2798us) server_negotiation.cc:407] Connection header received
0924 04:08:52.242437 (+ 1020us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:52.242441 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:52.242500 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:52.242585 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:52.244407 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:52.245252 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:52.247674 (+ 2422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:52.247988 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:52.248081 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:52.250592 (+ 2511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:52.250596 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:52.250599 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:52.250628 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:52.252521 (+ 1893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:52.253230 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:52.253237 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:52.253238 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:52.253342 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:52.253772 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:52.253776 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:52.253778 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:52.254018 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:52.254100 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:52.254807 (+ 707us) server_negotiation.cc:299] Negotiation successful
0924 04:08:52.254929 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 04:08:52.255990 7221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:52.238614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36703 (local address 127.0.98.61:43705)
0924 04:08:52.238729 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:52.238733 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:52.241243 (+ 2510us) server_negotiation.cc:407] Connection header received
0924 04:08:52.242858 (+ 1615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:52.242864 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:52.242940 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:52.242997 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:52.244400 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:52.245280 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:52.246949 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:52.247277 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:52.247364 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:52.251662 (+ 4298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:52.251667 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:52.251671 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:52.251702 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:52.253882 (+ 2180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:52.254323 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:52.254326 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:52.254328 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:52.254411 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:52.254685 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:52.254691 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:52.254696 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:52.254930 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:52.255017 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:52.255730 (+ 713us) server_negotiation.cc:299] Negotiation successful
0924 04:08:52.255851 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 04:08:52.256058 7468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:52.240302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35081 (local address 127.0.98.60:34051)
0924 04:08:52.240439 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:52.240444 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:52.242649 (+ 2205us) server_negotiation.cc:407] Connection header received
0924 04:08:52.242825 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:52.242830 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:52.242884 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:52.242968 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:52.244408 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:52.245252 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:52.247168 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:52.247501 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:52.247574 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:52.251685 (+ 4111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:52.251691 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:52.251698 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:52.251736 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:52.253600 (+ 1864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:52.254162 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:52.254168 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:52.254172 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:52.254263 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:52.254611 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:52.254616 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:52.254618 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:52.254862 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:52.254965 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:52.255665 (+ 700us) server_negotiation.cc:299] Negotiation successful
0924 04:08:52.255818 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 04:08:52.256919 7604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:52.238958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43705 (local address 127.0.98.2:36703)
0924 04:08:52.240833 (+ 1875us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:52.240900 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:52.242223 (+ 1323us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:52.243157 (+ 934us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:52.243162 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:52.243428 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:08:52.244248 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:52.244260 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:52.245453 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:52.245459 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:52.246825 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:52.246834 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:52.247546 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:52.247552 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:52.247750 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:52.248616 (+ 866us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:08:52.248642 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:08:52.251496 (+ 2854us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:08:52.253992 (+ 2496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:52.253995 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:52.253997 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:52.254222 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:52.254508 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:52.254512 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:52.254514 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:52.254587 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:52.255059 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:08:52.255062 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:08:52.255200 (+ 138us) client_negotiation.cc:769] Sending connection context
0924 04:08:52.255326 (+ 126us) client_negotiation.cc:240] Negotiation successful
0924 04:08:52.256256 (+ 930us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1620,"mutex_wait_us":881,"spinlock_wait_cycles":4864,"thread_start_us":686,"threads_started":1}
I20260924 04:08:52.256985 7602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:52.240201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34051 (local address 127.0.98.2:35081)
0924 04:08:52.242565 (+ 2364us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:52.242580 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:52.242717 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:52.243084 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:52.243090 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:52.243388 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:08:52.244236 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:52.244256 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:52.245437 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:52.245440 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:52.247059 (+ 1619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:52.247065 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:52.247718 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:52.247721 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:52.247793 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:52.249430 (+ 1637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:08:52.249462 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:08:52.251506 (+ 2044us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:08:52.253739 (+ 2233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:52.253746 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:52.253751 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:52.254037 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:52.254390 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:52.254395 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:52.254398 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:52.254493 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:52.255050 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:08:52.255055 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:08:52.255420 (+ 365us) client_negotiation.cc:769] Sending connection context
0924 04:08:52.256487 (+ 1067us) client_negotiation.cc:240] Negotiation successful
0924 04:08:52.256658 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2282,"mutex_wait_us":611,"spinlock_wait_cycles":18816,"thread_start_us":90,"threads_started":1}
I20260924 04:08:52.257184 7603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:52.238958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38949 (local address 127.0.98.2:57623)
0924 04:08:52.241319 (+ 2361us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:52.241334 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:52.242229 (+ 895us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:52.242789 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:52.242799 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:52.243388 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:08:52.244235 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:52.244256 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:52.245437 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:52.245440 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:52.247523 (+ 2083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:52.247536 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:52.248269 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:52.248273 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:52.248417 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:52.248872 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:08:52.248884 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:08:52.250386 (+ 1502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:08:52.252667 (+ 2281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:52.252674 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:52.252720 (+ 46us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:52.253105 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:52.253468 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:52.253471 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:52.253473 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:52.253656 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:52.254166 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:08:52.254173 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:08:52.254532 (+ 359us) client_negotiation.cc:769] Sending connection context
0924 04:08:52.255334 (+ 802us) client_negotiation.cc:240] Negotiation successful
0924 04:08:52.255760 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2267,"mutex_wait_us":1202,"spinlock_wait_cycles":69376,"thread_start_us":153,"threads_started":1}
I20260924 04:08:52.258617 7599 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38949
I20260924 04:08:52.258664 7600 heartbeater.cc:347] Connected to a master server at 127.0.98.61:43705
I20260924 04:08:52.258885 7600 heartbeater.cc:464] Registering TS with master...
I20260924 04:08:52.258957 7598 heartbeater.cc:347] Connected to a master server at 127.0.98.60:34051
I20260924 04:08:52.259028 7598 heartbeater.cc:464] Registering TS with master...
I20260924 04:08:52.258900 7599 heartbeater.cc:464] Registering TS with master...
I20260924 04:08:52.259450 7600 heartbeater.cc:511] Master 127.0.98.61:43705 requested a full tablet report, sending...
I20260924 04:08:52.259625 7599 heartbeater.cc:511] Master 127.0.98.62:38949 requested a full tablet report, sending...
I20260924 04:08:52.259456 7598 heartbeater.cc:511] Master 127.0.98.60:34051 requested a full tablet report, sending...
I20260924 04:08:52.260537 6499 ts_manager.cc:194] Re-registered known tserver with Master: 67def1cdfdb048899741175fab806933 (127.0.98.2:37085)
I20260924 04:08:52.260577 7098 ts_manager.cc:194] Registered new tserver with Master: 67def1cdfdb048899741175fab806933 (127.0.98.2:37085)
I20260924 04:08:52.261134 7274 ts_manager.cc:194] Registered new tserver with Master: 67def1cdfdb048899741175fab806933 (127.0.98.2:37085)
I20260924 04:08:52.261432 6499 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:57623
W20260924 04:08:52.361091 7605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:08:52.361349 7605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:08:52.361433 7605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:08:52.365800 7605 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:08:52.365885 7605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:08:52.365980 7605 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:08:52.371304 7605 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:33301
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=41897
--tserver_master_addrs=127.0.98.62:38949,127.0.98.61:43705,127.0.98.60:34051
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7605
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:08:52.372403 7605 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:08:52.373509 7605 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:08:52.383560 7605 server_base.cc:1034] running on GCE node
W20260924 04:08:52.383493 7613 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:08:52.383493 7610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:08:52.383499 7611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:08:52.384241 7605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:08:52.385076 7605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:08:52.386255 7605 hybrid_clock.cc:648] HybridClock initialized: now 1790222932386243 us; error 23 us; skew 500 ppm
Sep 24 04:08:52 dist-test-slave-00ld krb5kdc[6454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222932, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:08:52.389083 7605 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260924 04:08:52.389997 7605 webserver.cc:460] Webserver started at http://127.0.98.3:41897/ using document root <none> and password file <none>
I20260924 04:08:52.390740 7605 fs_manager.cc:359] Metadata directory not provided
I20260924 04:08:52.390820 7605 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:08:52.394696 7605 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:08:52.396741 7621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:52.397982 7605 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:08:52.398103 7605 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/ts-2/wal
uuid: "a5a609a5eb2a40cfaf3f6616900bd505"
format_stamp: "Formatted at 2026-09-24 04:08:51 on dist-test-slave-00ld"
server_key: "e7fb8776e1131fbf84c2d80d0dec85cd"
server_key_iv: "f3d3c2ae77ad6ddd6d2ddd061eabb61f"
server_key_version: "encryptionkey@0"
I20260924 04:08:52.398640 7605 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:08:52.408201 7605 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:08:52.411453 7605 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:08:52.411655 7605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:08:52.412261 7605 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:08:52.413384 7605 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:08:52.413455 7605 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:52.413517 7605 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:08:52.413556 7605 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:52.422580 7605 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33301
I20260924 04:08:52.422597 7734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33301 every 8 connection(s)
I20260924 04:08:52.423614 7605 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:08:52.423847 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 7605
Sep 24 04:08:52 dist-test-slave-00ld krb5kdc[6454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222932, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 24 04:08:52 dist-test-slave-00ld krb5kdc[6454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222932, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 04:08:52.438815 7221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:52.427642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42244 (local address 127.0.98.61:43705)
0924 04:08:52.427766 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:52.427770 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:52.427783 (+ 13us) server_negotiation.cc:407] Connection header received
0924 04:08:52.427827 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:52.427832 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:52.427871 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:52.427956 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:52.429775 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:52.430577 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:52.432101 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:52.432421 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:52.432502 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:52.434114 (+ 1612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:52.434117 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:52.434120 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:52.434143 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:52.436487 (+ 2344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:52.437001 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:52.437007 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:52.437012 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:52.437096 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:52.437513 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:52.437516 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:52.437518 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:52.437722 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:52.437841 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:52.438537 (+ 696us) server_negotiation.cc:299] Negotiation successful
0924 04:08:52.438648 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":56}
I20260924 04:08:52.440948 7752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:52.431231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34051 (local address 127.0.0.1:49636)
0924 04:08:52.435560 (+ 4329us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:52.435573 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:52.435659 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:52.437153 (+ 1494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:52.437156 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:52.437175 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:08:52.437288 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:52.437295 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:52.438236 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:52.438240 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:52.439358 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:52.439369 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:52.440480 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:52.440485 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:52.440606 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:52.440629 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 04:08:52.440755 (+ 126us) client_negotiation.cc:240] Negotiation successful
0924 04:08:52.440829 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4233,"thread_start_us":76,"threads_started":1}
Sep 24 04:08:52 dist-test-slave-00ld krb5kdc[6454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222932, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260924 04:08:52.441382 7753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:52.431524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49636 (local address 127.0.98.60:34051)
0924 04:08:52.436823 (+ 5299us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:52.436828 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:52.436840 (+ 12us) server_negotiation.cc:407] Connection header received
0924 04:08:52.436891 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:52.436894 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:52.436957 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:52.437049 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:08:52.437420 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:52.438121 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:52.439503 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:52.440144 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:52.440249 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:52.440416 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:52.441134 (+ 718us) server_negotiation.cc:299] Negotiation successful
0924 04:08:52.441217 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5226,"thread_start_us":66,"threads_started":1}
I20260924 04:08:52.442096 7286 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 90d32bb2c3d041d78f61550aa83a8a15 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260924 04:08:52.442617 7747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:52.426868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49624 (local address 127.0.98.60:34051)
0924 04:08:52.427327 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:52.427338 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:52.427662 (+ 324us) server_negotiation.cc:407] Connection header received
0924 04:08:52.427710 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:52.427713 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:52.427761 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:52.427841 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:52.428831 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:52.429660 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:52.430897 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:52.431237 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:52.431330 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:52.433396 (+ 2066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:52.433399 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:52.433402 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:52.433426 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:52.438846 (+ 5420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:52.439305 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:52.439309 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:52.439311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:52.439391 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:52.439685 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:52.439688 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:52.439690 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:52.439909 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:52.440037 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:52.441902 (+ 1865us) server_negotiation.cc:299] Negotiation successful
0924 04:08:52.442395 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":76,"threads_started":1}
I20260924 04:08:52.442677 7324 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90d32bb2c3d041d78f61550aa83a8a15 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd568f5711864cc4a75a0f2fef7236a2. Latest consensus state: current_term: 1 leader_uuid: "cd568f5711864cc4a75a0f2fef7236a2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cd568f5711864cc4a75a0f2fef7236a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38949 } } peers { permanent_uuid: "3c01afc87f8b4560b973c517eea256a2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43705 } attrs { promote: false } } peers { permanent_uuid: "90d32bb2c3d041d78f61550aa83a8a15" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 34051 } attrs { promote: true } } }
I20260924 04:08:52.442818 7324 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90d32bb2c3d041d78f61550aa83a8a15 [sys.catalog]: This master's current role is: LEARNER
I20260924 04:08:52.445233 7468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:52.425721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52877 (local address 127.0.98.60:34051)
0924 04:08:52.425846 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:52.425850 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:52.430825 (+ 4975us) server_negotiation.cc:407] Connection header received
0924 04:08:52.430995 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:52.430998 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:52.431048 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:52.431102 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:52.431682 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:52.432387 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:52.433556 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:52.433934 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:52.434019 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:52.440318 (+ 6299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:52.440321 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:52.440323 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:52.440345 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:52.442254 (+ 1909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:52.443310 (+ 1056us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:52.443313 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:52.443315 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:52.443395 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:52.443802 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:52.443835 (+ 33us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:52.443837 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:52.444040 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:52.444133 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:52.444947 (+ 814us) server_negotiation.cc:299] Negotiation successful
0924 04:08:52.445060 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 04:08:52.445498 6538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:52.425683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40741 (local address 127.0.98.62:38949)
0924 04:08:52.425815 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:52.425819 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:52.427272 (+ 1453us) server_negotiation.cc:407] Connection header received
0924 04:08:52.428488 (+ 1216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:52.428492 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:52.428546 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:52.428592 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:52.430297 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:52.431020 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:52.433398 (+ 2378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:52.433741 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:52.433829 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:52.439960 (+ 6131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:52.439966 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:52.439972 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:52.439996 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:52.442345 (+ 2349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:52.444137 (+ 1792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:52.444141 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:52.444143 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:52.444229 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:52.444545 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:52.444551 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:52.444553 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:52.444799 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:52.444890 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:52.445226 (+ 336us) server_negotiation.cc:299] Negotiation successful
0924 04:08:52.445347 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 04:08:52.447062 7749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:52.425999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34051 (local address 127.0.98.3:52877)
0924 04:08:52.430743 (+ 4744us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:52.430757 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:52.430894 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:52.431234 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:52.431237 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:52.431453 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:08:52.431561 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:52.431567 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:52.432513 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:52.432516 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:52.433408 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:52.433421 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:52.434163 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:52.434167 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:52.434325 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:52.435061 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:08:52.435080 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:08:52.440190 (+ 5110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:08:52.442761 (+ 2571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:52.442771 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:52.442796 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:52.443187 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:52.443505 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:52.443508 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:52.443510 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:52.443691 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:52.444202 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:08:52.444211 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:08:52.444616 (+ 405us) client_negotiation.cc:769] Sending connection context
0924 04:08:52.445436 (+ 820us) client_negotiation.cc:240] Negotiation successful
0924 04:08:52.445902 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4659,"mutex_wait_us":2035,"thread_start_us":111,"threads_started":1}
I20260924 04:08:52.447355 7746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:52.425999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38949 (local address 127.0.98.3:40741)
0924 04:08:52.427134 (+ 1135us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:52.427189 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:52.428256 (+ 1067us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:52.429000 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:52.429010 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:52.429480 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:08:52.430161 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:52.430179 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:52.431745 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:52.431751 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:52.433264 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:52.433275 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:52.434025 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:52.434029 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:52.434327 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:52.434930 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:08:52.434955 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:08:52.439806 (+ 4851us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:08:52.443732 (+ 3926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:52.443737 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:52.443741 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:52.444013 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:52.444340 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:52.444345 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:52.444347 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:52.444437 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:52.444949 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:08:52.444952 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:08:52.445048 (+ 96us) client_negotiation.cc:769] Sending connection context
0924 04:08:52.445646 (+ 598us) client_negotiation.cc:240] Negotiation successful
0924 04:08:52.446005 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":909,"mutex_wait_us":198,"spinlock_wait_cycles":7936,"thread_start_us":754,"threads_started":1}
I20260924 04:08:52.447674 7743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:52.426165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53622 (local address 127.0.98.62:38949)
0924 04:08:52.426461 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:52.426464 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:52.426851 (+ 387us) server_negotiation.cc:407] Connection header received
0924 04:08:52.426907 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:52.426911 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:52.426965 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:52.427044 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:52.428045 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:52.428854 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:52.430044 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:52.430381 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:52.430464 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:52.442039 (+ 11575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:52.442043 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:52.442046 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:52.442072 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:52.444649 (+ 2577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:52.445187 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:52.445190 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:52.445192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:52.445263 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:52.446016 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:52.446019 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:52.446021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:52.446203 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:52.446864 (+ 661us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:52.446938 (+ 74us) server_negotiation.cc:299] Negotiation successful
0924 04:08:52.447048 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":71,"threads_started":1}
I20260924 04:08:52.447520 7751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:52.428771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52789 (local address 127.0.98.61:43705)
0924 04:08:52.433058 (+ 4287us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:52.433062 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:52.434602 (+ 1540us) server_negotiation.cc:407] Connection header received
0924 04:08:52.434638 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:52.434641 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:52.434681 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:52.434762 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:52.435534 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:52.436404 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:52.437665 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:52.438041 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:52.438123 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:52.442621 (+ 4498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:52.442625 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:52.442628 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:52.442654 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:52.445321 (+ 2667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:52.445887 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:52.445890 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:52.445892 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:52.445977 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:52.446265 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:52.446268 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:52.446270 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:52.446486 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:52.446571 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:52.447206 (+ 635us) server_negotiation.cc:299] Negotiation successful
0924 04:08:52.447345 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4221,"thread_start_us":94,"threads_started":1}
I20260924 04:08:52.448290 7736 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38949
I20260924 04:08:52.448299 7735 heartbeater.cc:347] Connected to a master server at 127.0.98.60:34051
I20260924 04:08:52.448546 7736 heartbeater.cc:464] Registering TS with master...
I20260924 04:08:52.448546 7735 heartbeater.cc:464] Registering TS with master...
I20260924 04:08:52.447088 7750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:52.428434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43705 (local address 127.0.98.3:52789)
0924 04:08:52.434435 (+ 6001us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:52.434450 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:52.434532 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:52.434899 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:52.434901 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:52.435270 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:08:52.435395 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:52.435407 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:52.436547 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:52.436553 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:52.437522 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:52.437534 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:52.438282 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:52.438286 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:52.438393 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:52.438868 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:08:52.438886 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:08:52.442462 (+ 3576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:08:52.445443 (+ 2981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:52.445445 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:52.445447 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:52.445756 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:52.446071 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:52.446074 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:52.446076 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:52.446154 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:52.446611 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:08:52.446614 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:08:52.446707 (+ 93us) client_negotiation.cc:769] Sending connection context
0924 04:08:52.446799 (+ 92us) client_negotiation.cc:240] Negotiation successful
0924 04:08:52.446942 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5938,"thread_start_us":77,"threads_started":1}
I20260924 04:08:52.449575 7735 heartbeater.cc:511] Master 127.0.98.60:34051 requested a full tablet report, sending...
I20260924 04:08:52.449851 7737 heartbeater.cc:347] Connected to a master server at 127.0.98.61:43705
I20260924 04:08:52.449934 7737 heartbeater.cc:464] Registering TS with master...
I20260924 04:08:52.450073 7737 heartbeater.cc:511] Master 127.0.98.61:43705 requested a full tablet report, sending...
I20260924 04:08:52.450613 7736 heartbeater.cc:511] Master 127.0.98.62:38949 requested a full tablet report, sending...
I20260924 04:08:52.451220 6498 ts_manager.cc:194] Re-registered known tserver with Master: a5a609a5eb2a40cfaf3f6616900bd505 (127.0.98.3:33301)
I20260924 04:08:52.451922 6498 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40741
I20260924 04:08:52.450892 6499 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53622:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:08:52.454418 6499 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:08:52.455238 7274 ts_manager.cc:194] Registered new tserver with Master: a5a609a5eb2a40cfaf3f6616900bd505 (127.0.98.3:33301)
I20260924 04:08:52.457247 7098 ts_manager.cc:194] Registered new tserver with Master: a5a609a5eb2a40cfaf3f6616900bd505 (127.0.98.3:33301)
I20260924 04:08:52.458338 7615 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd568f5711864cc4a75a0f2fef7236a2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cd568f5711864cc4a75a0f2fef7236a2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cd568f5711864cc4a75a0f2fef7236a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38949 } } peers { permanent_uuid: "3c01afc87f8b4560b973c517eea256a2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43705 } attrs { promote: false } } peers { permanent_uuid: "90d32bb2c3d041d78f61550aa83a8a15" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 34051 } attrs { promote: true } } }
I20260924 04:08:52.458464 7615 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd568f5711864cc4a75a0f2fef7236a2 [sys.catalog]: This master's current role is: LEADER
I20260924 04:08:52.462065 7615 raft_consensus.cc:1050] T 00000000000000000000000000000000 P cd568f5711864cc4a75a0f2fef7236a2: attempting to promote NON_VOTER 90d32bb2c3d041d78f61550aa83a8a15 to VOTER
I20260924 04:08:52.462468 7615 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd568f5711864cc4a75a0f2fef7236a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cd568f5711864cc4a75a0f2fef7236a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38949 } } peers { permanent_uuid: "3c01afc87f8b4560b973c517eea256a2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43705 } attrs { promote: false } } peers { permanent_uuid: "90d32bb2c3d041d78f61550aa83a8a15" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34051 } attrs { promote: false } }
I20260924 04:08:52.463558 7286 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 90d32bb2c3d041d78f61550aa83a8a15 [term 1 LEARNER]: Refusing update from remote peer cd568f5711864cc4a75a0f2fef7236a2: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260924 04:08:52.463605 7109 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3c01afc87f8b4560b973c517eea256a2 [term 1 FOLLOWER]: Refusing update from remote peer cd568f5711864cc4a75a0f2fef7236a2: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260924 04:08:52.465103 7756 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cd568f5711864cc4a75a0f2fef7236a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c01afc87f8b4560b973c517eea256a2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43705 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260924 04:08:52.465262 7757 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cd568f5711864cc4a75a0f2fef7236a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d32bb2c3d041d78f61550aa83a8a15" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34051 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260924 04:08:52.466912 7756 raft_consensus.cc:2932] T 00000000000000000000000000000000 P cd568f5711864cc4a75a0f2fef7236a2 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 90d32bb2c3d041d78f61550aa83a8a15 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cd568f5711864cc4a75a0f2fef7236a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38949 } } peers { permanent_uuid: "3c01afc87f8b4560b973c517eea256a2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43705 } attrs { promote: false } } peers { permanent_uuid: "90d32bb2c3d041d78f61550aa83a8a15" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34051 } attrs { promote: false } } }
I20260924 04:08:52.467419 7109 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3c01afc87f8b4560b973c517eea256a2 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 90d32bb2c3d041d78f61550aa83a8a15 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cd568f5711864cc4a75a0f2fef7236a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38949 } } peers { permanent_uuid: "3c01afc87f8b4560b973c517eea256a2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43705 } attrs { promote: false } } peers { permanent_uuid: "90d32bb2c3d041d78f61550aa83a8a15" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34051 } attrs { promote: false } } }
I20260924 04:08:52.467425 7286 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 90d32bb2c3d041d78f61550aa83a8a15 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 90d32bb2c3d041d78f61550aa83a8a15 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cd568f5711864cc4a75a0f2fef7236a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38949 } } peers { permanent_uuid: "3c01afc87f8b4560b973c517eea256a2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43705 } attrs { promote: false } } peers { permanent_uuid: "90d32bb2c3d041d78f61550aa83a8a15" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34051 } attrs { promote: false } } }
I20260924 04:08:52.468442 7756 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd568f5711864cc4a75a0f2fef7236a2 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "cd568f5711864cc4a75a0f2fef7236a2" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cd568f5711864cc4a75a0f2fef7236a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38949 } } peers { permanent_uuid: "3c01afc87f8b4560b973c517eea256a2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43705 } attrs { promote: false } } peers { permanent_uuid: "90d32bb2c3d041d78f61550aa83a8a15" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34051 } attrs { promote: false } } }
I20260924 04:08:52.468537 7756 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd568f5711864cc4a75a0f2fef7236a2 [sys.catalog]: This master's current role is: LEADER
I20260924 04:08:52.468704 7756 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd568f5711864cc4a75a0f2fef7236a2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cd568f5711864cc4a75a0f2fef7236a2" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cd568f5711864cc4a75a0f2fef7236a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38949 } } peers { permanent_uuid: "3c01afc87f8b4560b973c517eea256a2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43705 } attrs { promote: false } } peers { permanent_uuid: "90d32bb2c3d041d78f61550aa83a8a15" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34051 } attrs { promote: false } } }
I20260924 04:08:52.468791 7756 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd568f5711864cc4a75a0f2fef7236a2 [sys.catalog]: This master's current role is: LEADER
I20260924 04:08:52.468924 7756 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd568f5711864cc4a75a0f2fef7236a2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cd568f5711864cc4a75a0f2fef7236a2" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cd568f5711864cc4a75a0f2fef7236a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38949 } } peers { permanent_uuid: "3c01afc87f8b4560b973c517eea256a2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43705 } attrs { promote: false } } peers { permanent_uuid: "90d32bb2c3d041d78f61550aa83a8a15" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34051 } attrs { promote: false } } }
I20260924 04:08:52.468997 7756 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd568f5711864cc4a75a0f2fef7236a2 [sys.catalog]: This master's current role is: LEADER
I20260924 04:08:52.469333 7765 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c01afc87f8b4560b973c517eea256a2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cd568f5711864cc4a75a0f2fef7236a2" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cd568f5711864cc4a75a0f2fef7236a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38949 } } peers { permanent_uuid: "3c01afc87f8b4560b973c517eea256a2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43705 } attrs { promote: false } } peers { permanent_uuid: "90d32bb2c3d041d78f61550aa83a8a15" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34051 } attrs { promote: false } } }
I20260924 04:08:52.469456 7765 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c01afc87f8b4560b973c517eea256a2 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:08:52.469798 7324 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90d32bb2c3d041d78f61550aa83a8a15 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cd568f5711864cc4a75a0f2fef7236a2" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cd568f5711864cc4a75a0f2fef7236a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38949 } } peers { permanent_uuid: "3c01afc87f8b4560b973c517eea256a2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43705 } attrs { promote: false } } peers { permanent_uuid: "90d32bb2c3d041d78f61550aa83a8a15" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34051 } attrs { promote: false } } }
I20260924 04:08:52.469955 7324 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90d32bb2c3d041d78f61550aa83a8a15 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:08:52.478372 7752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:52.472432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37085 (local address 127.0.0.1:42982)
0924 04:08:52.472609 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:52.472627 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:52.472735 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:52.473645 (+ 910us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:52.473649 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:52.473670 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:08:52.473779 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:52.473787 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:52.475455 (+ 1668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:52.475458 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:52.476530 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:52.476539 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:52.477917 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:52.477923 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:52.478014 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:52.478042 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 04:08:52.478095 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 04:08:52.478179 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":3}
I20260924 04:08:52.478566 7767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:52.473013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44685 (local address 127.0.0.1:54070)
0924 04:08:52.473764 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:52.473777 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:52.473869 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:52.474458 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:52.474461 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:52.474476 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:08:52.474558 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:52.474565 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:52.475936 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:52.475939 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:52.476843 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:52.476852 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:52.478090 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:52.478094 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:52.478258 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:52.478280 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 04:08:52.478354 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 04:08:52.478430 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":700,"mutex_wait_us":55,"thread_start_us":106,"threads_started":1}
I20260924 04:08:52.478606 7770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:52.473306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45942 (local address 127.0.98.3:33301)
0924 04:08:52.473596 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:52.473608 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:52.473628 (+ 20us) server_negotiation.cc:407] Connection header received
0924 04:08:52.473697 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:52.473702 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:52.473940 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:52.474458 (+ 518us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:08:52.474582 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:52.475746 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:52.477235 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:52.477975 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:52.478075 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:52.478205 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:52.478369 (+ 164us) server_negotiation.cc:299] Negotiation successful
0924 04:08:52.478470 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":122,"threads_started":1}
I20260924 04:08:52.478961 7768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:52.472850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33301 (local address 127.0.0.1:45942)
0924 04:08:52.473240 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:52.473254 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:52.473344 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:52.474093 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:52.474095 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:52.474110 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:08:52.474198 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:52.474205 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:52.475867 (+ 1662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:52.475870 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:52.477099 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:52.477109 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:52.478079 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:52.478083 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:52.478202 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:52.478252 (+ 50us) client_negotiation.cc:769] Sending connection context
0924 04:08:52.478320 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 04:08:52.478390 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"mutex_wait_us":106,"thread_start_us":105,"threads_started":1}
I20260924 04:08:52.479009 7769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:52.473190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54070 (local address 127.0.98.1:44685)
0924 04:08:52.473638 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:52.473649 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:52.473896 (+ 247us) server_negotiation.cc:407] Connection header received
0924 04:08:52.473976 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:52.473980 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:52.474154 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:52.474299 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:08:52.474687 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:52.475812 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:52.476966 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:52.477743 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:52.477852 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:52.478028 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:52.478761 (+ 733us) server_negotiation.cc:299] Negotiation successful
0924 04:08:52.478868 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":156,"threads_started":1}
I20260924 04:08:52.479125 7766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:52.472657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42982 (local address 127.0.98.2:37085)
0924 04:08:52.473115 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:52.473128 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:52.473151 (+ 23us) server_negotiation.cc:407] Connection header received
0924 04:08:52.473232 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:52.473238 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:52.473477 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:52.473990 (+ 513us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:08:52.474154 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:52.475339 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:52.476917 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:52.477786 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:52.478658 (+ 872us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:52.478815 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:52.478863 (+ 48us) server_negotiation.cc:299] Negotiation successful
0924 04:08:52.478947 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":136,"threads_started":1}
I20260924 04:08:52.481839 7395 tablet_service.cc:1467] Processing CreateTablet for tablet e7028b69a2ad4604acdbf6cdc9fcd6c8 (DEFAULT_TABLE table=test-table [id=a67e810ad1cb49ee836397a8284c8bb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:08:52.481829 7532 tablet_service.cc:1467] Processing CreateTablet for tablet e7028b69a2ad4604acdbf6cdc9fcd6c8 (DEFAULT_TABLE table=test-table [id=a67e810ad1cb49ee836397a8284c8bb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:08:52.482072 7669 tablet_service.cc:1467] Processing CreateTablet for tablet e7028b69a2ad4604acdbf6cdc9fcd6c8 (DEFAULT_TABLE table=test-table [id=a67e810ad1cb49ee836397a8284c8bb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:08:52.482839 7395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7028b69a2ad4604acdbf6cdc9fcd6c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:08:52.482856 7532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7028b69a2ad4604acdbf6cdc9fcd6c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:08:52.482856 7669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7028b69a2ad4604acdbf6cdc9fcd6c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:08:52.489753 7771 tablet_bootstrap.cc:492] T e7028b69a2ad4604acdbf6cdc9fcd6c8 P dc6019f5f5474832b90b9150dcc4b0b3: Bootstrap starting.
I20260924 04:08:52.491014 7773 tablet_bootstrap.cc:492] T e7028b69a2ad4604acdbf6cdc9fcd6c8 P 67def1cdfdb048899741175fab806933: Bootstrap starting.
I20260924 04:08:52.491647 7772 tablet_bootstrap.cc:492] T e7028b69a2ad4604acdbf6cdc9fcd6c8 P a5a609a5eb2a40cfaf3f6616900bd505: Bootstrap starting.
I20260924 04:08:52.492671 7771 tablet_bootstrap.cc:654] T e7028b69a2ad4604acdbf6cdc9fcd6c8 P dc6019f5f5474832b90b9150dcc4b0b3: Neither blocks nor log segments found. Creating new log.
I20260924 04:08:52.493129 7773 tablet_bootstrap.cc:654] T e7028b69a2ad4604acdbf6cdc9fcd6c8 P 67def1cdfdb048899741175fab806933: Neither blocks nor log segments found. Creating new log.
I20260924 04:08:52.493492 7772 tablet_bootstrap.cc:654] T e7028b69a2ad4604acdbf6cdc9fcd6c8 P a5a609a5eb2a40cfaf3f6616900bd505: Neither blocks nor log segments found. Creating new log.
I20260924 04:08:52.493600 7771 log.cc:824] T e7028b69a2ad4604acdbf6cdc9fcd6c8 P dc6019f5f5474832b90b9150dcc4b0b3: Log is configured to *not* fsync() on all Append() calls
I20260924 04:08:52.493860 7773 log.cc:824] T e7028b69a2ad4604acdbf6cdc9fcd6c8 P 67def1cdfdb048899741175fab806933: Log is configured to *not* fsync() on all Append() calls
I20260924 04:08:52.494241 7772 log.cc:824] T e7028b69a2ad4604acdbf6cdc9fcd6c8 P a5a609a5eb2a40cfaf3f6616900bd505: Log is configured to *not* fsync() on all Append() calls
I20260924 04:08:52.495605 7771 tablet_bootstrap.cc:492] T e7028b69a2ad4604acdbf6cdc9fcd6c8 P dc6019f5f5474832b90b9150dcc4b0b3: No bootstrap required, opened a new log
I20260924 04:08:52.495719 7773 tablet_bootstrap.cc:492] T e7028b69a2ad4604acdbf6cdc9fcd6c8 P 67def1cdfdb048899741175fab806933: No bootstrap required, opened a new log
I20260924 04:08:52.495851 7771 ts_tablet_manager.cc:1397] T e7028b69a2ad4604acdbf6cdc9fcd6c8 P dc6019f5f5474832b90b9150dcc4b0b3: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 04:08:52.495916 7773 ts_tablet_manager.cc:1397] T e7028b69a2ad4604acdbf6cdc9fcd6c8 P 67def1cdfdb048899741175fab806933: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 04:08:52.496031 7772 tablet_bootstrap.cc:492] T e7028b69a2ad4604acdbf6cdc9fcd6c8 P a5a609a5eb2a40cfaf3f6616900bd505: No bootstrap required, opened a new log
I20260924 04:08:52.496196 7772 ts_tablet_manager.cc:1397] T e7028b69a2ad4604acdbf6cdc9fcd6c8 P a5a609a5eb2a40cfaf3f6616900bd505: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 04:08:52.499799 7771 raft_consensus.cc:348] T e7028b69a2ad4604acdbf6cdc9fcd6c8 P dc6019f5f5474832b90b9150dcc4b0b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67def1cdfdb048899741175fab806933" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37085 } } peers { permanent_uuid: "a5a609a5eb2a40cfaf3f6616900bd505" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33301 } } peers { permanent_uuid: "dc6019f5f5474832b90b9150dcc4b0b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44685 } }
I20260924 04:08:52.500059 7771 raft_consensus.cc:374] T e7028b69a2ad4604acdbf6cdc9fcd6c8 P dc6019f5f5474832b90b9150dcc4b0b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:08:52.500154 7771 raft_consensus.cc:729] T e7028b69a2ad4604acdbf6cdc9fcd6c8 P dc6019f5f5474832b90b9150dcc4b0b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc6019f5f5474832b90b9150dcc4b0b3, State: Initialized, Role: FOLLOWER
I20260924 04:08:52.500656 7771 consensus_queue.cc:260] T e7028b69a2ad4604acdbf6cdc9fcd6c8 P dc6019f5f5474832b90b9150dcc4b0b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67def1cdfdb048899741175fab806933" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37085 } } peers { permanent_uuid: "a5a609a5eb2a40cfaf3f6616900bd505" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33301 } } peers { permanent_uuid: "dc6019f5f5474832b90b9150dcc4b0b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44685 } }
I20260924 04:08:52.501075 7773 raft_consensus.cc:348] T e7028b69a2ad4604acdbf6cdc9fcd6c8 P 67def1cdfdb048899741175fab806933 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67def1cdfdb048899741175fab806933" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37085 } } peers { permanent_uuid: "a5a609a5eb2a40cfaf3f6616900bd505" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33301 } } peers { permanent_uuid: "dc6019f5f5474832b90b9150dcc4b0b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44685 } }
I20260924 04:08:52.501367 7773 raft_consensus.cc:374] T e7028b69a2ad4604acdbf6cdc9fcd6c8 P 67def1cdfdb048899741175fab806933 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:08:52.501422 7773 raft_consensus.cc:729] T e7028b69a2ad4604acdbf6cdc9fcd6c8 P 67def1cdfdb048899741175fab806933 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67def1cdfdb048899741175fab806933, State: Initialized, Role: FOLLOWER
I20260924 04:08:52.501385 7772 raft_consensus.cc:348] T e7028b69a2ad4604acdbf6cdc9fcd6c8 P a5a609a5eb2a40cfaf3f6616900bd505 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67def1cdfdb048899741175fab806933" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37085 } } peers { permanent_uuid: "a5a609a5eb2a40cfaf3f6616900bd505" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33301 } } peers { permanent_uuid: "dc6019f5f5474832b90b9150dcc4b0b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44685 } }
I20260924 04:08:52.501644 7772 raft_consensus.cc:374] T e7028b69a2ad4604acdbf6cdc9fcd6c8 P a5a609a5eb2a40cfaf3f6616900bd505 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:08:52.501690 7772 raft_consensus.cc:729] T e7028b69a2ad4604acdbf6cdc9fcd6c8 P a5a609a5eb2a40cfaf3f6616900bd505 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5a609a5eb2a40cfaf3f6616900bd505, State: Initialized, Role: FOLLOWER
I20260924 04:08:52.501896 7771 ts_tablet_manager.cc:1428] T e7028b69a2ad4604acdbf6cdc9fcd6c8 P dc6019f5f5474832b90b9150dcc4b0b3: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260924 04:08:52.501844 7773 consensus_queue.cc:260] T e7028b69a2ad4604acdbf6cdc9fcd6c8 P 67def1cdfdb048899741175fab806933 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67def1cdfdb048899741175fab806933" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37085 } } peers { permanent_uuid: "a5a609a5eb2a40cfaf3f6616900bd505" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33301 } } peers { permanent_uuid: "dc6019f5f5474832b90b9150dcc4b0b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44685 } }
I20260924 04:08:52.502082 7772 consensus_queue.cc:260] T e7028b69a2ad4604acdbf6cdc9fcd6c8 P a5a609a5eb2a40cfaf3f6616900bd505 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67def1cdfdb048899741175fab806933" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37085 } } peers { permanent_uuid: "a5a609a5eb2a40cfaf3f6616900bd505" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33301 } } peers { permanent_uuid: "dc6019f5f5474832b90b9150dcc4b0b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44685 } }
I20260924 04:08:52.502468 7462 heartbeater.cc:503] Master 127.0.98.62:38949 was elected leader, sending a full tablet report...
I20260924 04:08:52.502753 7773 ts_tablet_manager.cc:1428] T e7028b69a2ad4604acdbf6cdc9fcd6c8 P 67def1cdfdb048899741175fab806933: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260924 04:08:52.503090 7772 ts_tablet_manager.cc:1428] T e7028b69a2ad4604acdbf6cdc9fcd6c8 P a5a609a5eb2a40cfaf3f6616900bd505: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260924 04:08:52.504742 7599 heartbeater.cc:503] Master 127.0.98.62:38949 was elected leader, sending a full tablet report...
I20260924 04:08:52.505252 7736 heartbeater.cc:503] Master 127.0.98.62:38949 was elected leader, sending a full tablet report...
W20260924 04:08:52.537479 7464 tablet.cc:2369] T e7028b69a2ad4604acdbf6cdc9fcd6c8 P dc6019f5f5474832b90b9150dcc4b0b3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 04:08:52.675894 7738 tablet.cc:2369] T e7028b69a2ad4604acdbf6cdc9fcd6c8 P a5a609a5eb2a40cfaf3f6616900bd505: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 04:08:52.738620 7601 tablet.cc:2369] T e7028b69a2ad4604acdbf6cdc9fcd6c8 P 67def1cdfdb048899741175fab806933: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:08:52.949498 7778 raft_consensus.cc:482] T e7028b69a2ad4604acdbf6cdc9fcd6c8 P 67def1cdfdb048899741175fab806933 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:08:52.949736 7778 raft_consensus.cc:504] T e7028b69a2ad4604acdbf6cdc9fcd6c8 P 67def1cdfdb048899741175fab806933 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67def1cdfdb048899741175fab806933" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37085 } } peers { permanent_uuid: "a5a609a5eb2a40cfaf3f6616900bd505" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33301 } } peers { permanent_uuid: "dc6019f5f5474832b90b9150dcc4b0b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44685 } }
I20260924 04:08:52.950892 7778 leader_election.cc:288] T e7028b69a2ad4604acdbf6cdc9fcd6c8 P 67def1cdfdb048899741175fab806933 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5a609a5eb2a40cfaf3f6616900bd505 (127.0.98.3:33301), dc6019f5f5474832b90b9150dcc4b0b3 (127.0.98.1:44685)
I20260924 04:08:52.954849 7769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:52.951285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39411 (local address 127.0.98.1:44685)
0924 04:08:52.951398 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:52.951405 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:52.951601 (+ 196us) server_negotiation.cc:407] Connection header received
0924 04:08:52.951761 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:52.951764 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:52.951822 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:52.951880 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:08:52.952177 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:52.952732 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:52.953882 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:52.954290 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:52.954369 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:52.954459 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:52.954634 (+ 175us) server_negotiation.cc:299] Negotiation successful
0924 04:08:52.954713 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 04:08:52.954892 7770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:52.951234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35383 (local address 127.0.98.3:33301)
0924 04:08:52.951380 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:52.951385 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:52.951635 (+ 250us) server_negotiation.cc:407] Connection header received
0924 04:08:52.951806 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:52.951808 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:52.951851 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:52.951950 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:08:52.952242 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:52.952817 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:52.953942 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:52.954310 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:52.954439 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:52.954501 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:52.954703 (+ 202us) server_negotiation.cc:299] Negotiation successful
0924 04:08:52.954768 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 04:08:52.954916 7780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:52.951157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44685 (local address 127.0.98.2:39411)
0924 04:08:52.951505 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:52.951517 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:52.951642 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:52.951916 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:52.951921 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:52.951943 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:08:52.952069 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:52.952076 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:52.952831 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:52.952835 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:52.953767 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:52.953776 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:52.954413 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:52.954417 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:52.954505 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:52.954533 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 04:08:52.954607 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 04:08:52.954726 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"mutex_wait_us":35,"thread_start_us":88,"threads_started":1}
I20260924 04:08:52.955358 7781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:52.951160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33301 (local address 127.0.98.2:35383)
0924 04:08:52.951514 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:52.951525 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:52.951660 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:52.951983 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:52.951986 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:52.952002 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:08:52.952115 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:52.952139 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:52.952923 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:52.952926 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:52.953796 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:52.953805 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:52.954456 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:52.954459 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:52.954558 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:52.954598 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 04:08:52.954676 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 04:08:52.954759 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"mutex_wait_us":119,"thread_start_us":62,"threads_started":1}
I20260924 04:08:52.955719 7415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7028b69a2ad4604acdbf6cdc9fcd6c8" candidate_uuid: "67def1cdfdb048899741175fab806933" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc6019f5f5474832b90b9150dcc4b0b3" is_pre_election: true
I20260924 04:08:52.955957 7689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7028b69a2ad4604acdbf6cdc9fcd6c8" candidate_uuid: "67def1cdfdb048899741175fab806933" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5a609a5eb2a40cfaf3f6616900bd505" is_pre_election: true
I20260924 04:08:52.956082 7415 raft_consensus.cc:2454] T e7028b69a2ad4604acdbf6cdc9fcd6c8 P dc6019f5f5474832b90b9150dcc4b0b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67def1cdfdb048899741175fab806933 in term 0.
I20260924 04:08:52.956244 7689 raft_consensus.cc:2454] T e7028b69a2ad4604acdbf6cdc9fcd6c8 P a5a609a5eb2a40cfaf3f6616900bd505 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67def1cdfdb048899741175fab806933 in term 0.
I20260924 04:08:52.956666 7488 leader_election.cc:302] T e7028b69a2ad4604acdbf6cdc9fcd6c8 P 67def1cdfdb048899741175fab806933 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 67def1cdfdb048899741175fab806933, a5a609a5eb2a40cfaf3f6616900bd505, dc6019f5f5474832b90b9150dcc4b0b3; no voters:
I20260924 04:08:52.957038 7778 raft_consensus.cc:2781] T e7028b69a2ad4604acdbf6cdc9fcd6c8 P 67def1cdfdb048899741175fab806933 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:08:52.957144 7778 raft_consensus.cc:482] T e7028b69a2ad4604acdbf6cdc9fcd6c8 P 67def1cdfdb048899741175fab806933 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:08:52.957211 7778 raft_consensus.cc:3037] T e7028b69a2ad4604acdbf6cdc9fcd6c8 P 67def1cdfdb048899741175fab806933 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:08:52.958268 7778 raft_consensus.cc:504] T e7028b69a2ad4604acdbf6cdc9fcd6c8 P 67def1cdfdb048899741175fab806933 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67def1cdfdb048899741175fab806933" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37085 } } peers { permanent_uuid: "a5a609a5eb2a40cfaf3f6616900bd505" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33301 } } peers { permanent_uuid: "dc6019f5f5474832b90b9150dcc4b0b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44685 } }
I20260924 04:08:52.958618 7778 leader_election.cc:288] T e7028b69a2ad4604acdbf6cdc9fcd6c8 P 67def1cdfdb048899741175fab806933 [CANDIDATE]: Term 1 election: Requested vote from peers a5a609a5eb2a40cfaf3f6616900bd505 (127.0.98.3:33301), dc6019f5f5474832b90b9150dcc4b0b3 (127.0.98.1:44685)
I20260924 04:08:52.958982 7689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7028b69a2ad4604acdbf6cdc9fcd6c8" candidate_uuid: "67def1cdfdb048899741175fab806933" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5a609a5eb2a40cfaf3f6616900bd505"
I20260924 04:08:52.959112 7689 raft_consensus.cc:3037] T e7028b69a2ad4604acdbf6cdc9fcd6c8 P a5a609a5eb2a40cfaf3f6616900bd505 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:08:52.960023 7689 raft_consensus.cc:2454] T e7028b69a2ad4604acdbf6cdc9fcd6c8 P a5a609a5eb2a40cfaf3f6616900bd505 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67def1cdfdb048899741175fab806933 in term 1.
I20260924 04:08:52.960304 7486 leader_election.cc:302] T e7028b69a2ad4604acdbf6cdc9fcd6c8 P 67def1cdfdb048899741175fab806933 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67def1cdfdb048899741175fab806933, a5a609a5eb2a40cfaf3f6616900bd505; no voters:
I20260924 04:08:52.960342 7415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7028b69a2ad4604acdbf6cdc9fcd6c8" candidate_uuid: "67def1cdfdb048899741175fab806933" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc6019f5f5474832b90b9150dcc4b0b3"
I20260924 04:08:52.960513 7778 raft_consensus.cc:2781] T e7028b69a2ad4604acdbf6cdc9fcd6c8 P 67def1cdfdb048899741175fab806933 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:08:52.960480 7415 raft_consensus.cc:3037] T e7028b69a2ad4604acdbf6cdc9fcd6c8 P dc6019f5f5474832b90b9150dcc4b0b3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:08:52.960719 7778 raft_consensus.cc:686] T e7028b69a2ad4604acdbf6cdc9fcd6c8 P 67def1cdfdb048899741175fab806933 [term 1 LEADER]: Becoming Leader. State: Replica: 67def1cdfdb048899741175fab806933, State: Running, Role: LEADER
I20260924 04:08:52.961151 7778 consensus_queue.cc:237] T e7028b69a2ad4604acdbf6cdc9fcd6c8 P 67def1cdfdb048899741175fab806933 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67def1cdfdb048899741175fab806933" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37085 } } peers { permanent_uuid: "a5a609a5eb2a40cfaf3f6616900bd505" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33301 } } peers { permanent_uuid: "dc6019f5f5474832b90b9150dcc4b0b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44685 } }
I20260924 04:08:52.961828 7415 raft_consensus.cc:2454] T e7028b69a2ad4604acdbf6cdc9fcd6c8 P dc6019f5f5474832b90b9150dcc4b0b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67def1cdfdb048899741175fab806933 in term 1.
I20260924 04:08:52.964505 6499 catalog_manager.cc:5495] T e7028b69a2ad4604acdbf6cdc9fcd6c8 P 67def1cdfdb048899741175fab806933 reported cstate change: term changed from 0 to 1, leader changed from <none> to 67def1cdfdb048899741175fab806933 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "67def1cdfdb048899741175fab806933" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67def1cdfdb048899741175fab806933" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5a609a5eb2a40cfaf3f6616900bd505" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc6019f5f5474832b90b9150dcc4b0b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44685 } health_report { overall_health: UNKNOWN } } }
I20260924 04:08:53.036780 7786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:53.032945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42994 (local address 127.0.98.2:37085)
0924 04:08:53.033156 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:53.033159 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:53.033172 (+ 13us) server_negotiation.cc:407] Connection header received
0924 04:08:53.033246 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:53.033248 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:53.033301 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:53.033419 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:08:53.033649 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:53.034153 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:53.035137 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:53.035400 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:53.035526 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:53.035877 (+ 351us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:08:53.036308 (+ 431us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:08:53.036460 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:53.036504 (+ 44us) server_negotiation.cc:299] Negotiation successful
0924 04:08:53.036563 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":74,"threads_started":1}
I20260924 04:08:53.043150 7689 raft_consensus.cc:1261] T e7028b69a2ad4604acdbf6cdc9fcd6c8 P a5a609a5eb2a40cfaf3f6616900bd505 [term 1 FOLLOWER]: Refusing update from remote peer 67def1cdfdb048899741175fab806933: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:08:53.043150 7415 raft_consensus.cc:1261] T e7028b69a2ad4604acdbf6cdc9fcd6c8 P dc6019f5f5474832b90b9150dcc4b0b3 [term 1 FOLLOWER]: Refusing update from remote peer 67def1cdfdb048899741175fab806933: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:08:53.043949 7782 consensus_queue.cc:1035] T e7028b69a2ad4604acdbf6cdc9fcd6c8 P 67def1cdfdb048899741175fab806933 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc6019f5f5474832b90b9150dcc4b0b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:08:53.044123 7778 consensus_queue.cc:1035] T e7028b69a2ad4604acdbf6cdc9fcd6c8 P 67def1cdfdb048899741175fab806933 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5a609a5eb2a40cfaf3f6616900bd505" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:08:53.053112 7790 mvcc.cc:205] Tried to move back new op lower bound from 7332753133737398272 to 7332753133412245504. Current Snapshot: MvccSnapshot[applied={T|T < 7332753133737398272}]
I20260924 04:08:53.055644 7791 mvcc.cc:205] Tried to move back new op lower bound from 7332753133737398272 to 7332753133412245504. Current Snapshot: MvccSnapshot[applied={T|T < 7332753133737398272}]
I20260924 04:08:53.063153 6498 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53622:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:08:53.063331 6498 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53622:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:08:53.068351 6498 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cd568f5711864cc4a75a0f2fef7236a2: Sending DeleteTablet for 3 replicas of tablet e7028b69a2ad4604acdbf6cdc9fcd6c8
I20260924 04:08:53.069222 7395 tablet_service.cc:1514] Processing DeleteTablet for tablet e7028b69a2ad4604acdbf6cdc9fcd6c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:08:53 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54070
I20260924 04:08:53.069329 7532 tablet_service.cc:1514] Processing DeleteTablet for tablet e7028b69a2ad4604acdbf6cdc9fcd6c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:08:53 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42982
I20260924 04:08:53.069761 7801 tablet_replica.cc:330] T e7028b69a2ad4604acdbf6cdc9fcd6c8 P dc6019f5f5474832b90b9150dcc4b0b3: stopping tablet replica
I20260924 04:08:53.069759 7669 tablet_service.cc:1514] Processing DeleteTablet for tablet e7028b69a2ad4604acdbf6cdc9fcd6c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:08:53 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45942
I20260924 04:08:53.070194 7803 tablet_replica.cc:330] T e7028b69a2ad4604acdbf6cdc9fcd6c8 P a5a609a5eb2a40cfaf3f6616900bd505: stopping tablet replica
I20260924 04:08:53.070201 7801 raft_consensus.cc:2229] T e7028b69a2ad4604acdbf6cdc9fcd6c8 P dc6019f5f5474832b90b9150dcc4b0b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:08:53.070498 7803 raft_consensus.cc:2229] T e7028b69a2ad4604acdbf6cdc9fcd6c8 P a5a609a5eb2a40cfaf3f6616900bd505 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:08:53.070690 7801 raft_consensus.cc:2258] T e7028b69a2ad4604acdbf6cdc9fcd6c8 P dc6019f5f5474832b90b9150dcc4b0b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:08:53.070760 7803 raft_consensus.cc:2258] T e7028b69a2ad4604acdbf6cdc9fcd6c8 P a5a609a5eb2a40cfaf3f6616900bd505 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:08:53.072117 7801 ts_tablet_manager.cc:1905] T e7028b69a2ad4604acdbf6cdc9fcd6c8 P dc6019f5f5474832b90b9150dcc4b0b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:08:53.072117 7803 ts_tablet_manager.cc:1905] T e7028b69a2ad4604acdbf6cdc9fcd6c8 P a5a609a5eb2a40cfaf3f6616900bd505: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:08:53.072770 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 6467
I20260924 04:08:53.076166 7803 ts_tablet_manager.cc:1918] T e7028b69a2ad4604acdbf6cdc9fcd6c8 P a5a609a5eb2a40cfaf3f6616900bd505: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:08:53.076166 7801 ts_tablet_manager.cc:1918] T e7028b69a2ad4604acdbf6cdc9fcd6c8 P dc6019f5f5474832b90b9150dcc4b0b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:08:53.076272 7803 log.cc:1196] T e7028b69a2ad4604acdbf6cdc9fcd6c8 P a5a609a5eb2a40cfaf3f6616900bd505: Deleting WAL directory at /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/ts-2/wal/wals/e7028b69a2ad4604acdbf6cdc9fcd6c8
I20260924 04:08:53.076403 7801 log.cc:1196] T e7028b69a2ad4604acdbf6cdc9fcd6c8 P dc6019f5f5474832b90b9150dcc4b0b3: Deleting WAL directory at /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/ts-0/wal/wals/e7028b69a2ad4604acdbf6cdc9fcd6c8
I20260924 04:08:53.076606 7803 ts_tablet_manager.cc:1939] T e7028b69a2ad4604acdbf6cdc9fcd6c8 P a5a609a5eb2a40cfaf3f6616900bd505: Deleting consensus metadata
I20260924 04:08:53.076877 7801 ts_tablet_manager.cc:1939] T e7028b69a2ad4604acdbf6cdc9fcd6c8 P dc6019f5f5474832b90b9150dcc4b0b3: Deleting consensus metadata
I20260924 04:08:53.077925 7802 tablet_replica.cc:330] T e7028b69a2ad4604acdbf6cdc9fcd6c8 P 67def1cdfdb048899741175fab806933: stopping tablet replica
I20260924 04:08:53.078275 7802 raft_consensus.cc:2229] T e7028b69a2ad4604acdbf6cdc9fcd6c8 P 67def1cdfdb048899741175fab806933 [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:08:53.078660 7802 raft_consensus.cc:2258] T e7028b69a2ad4604acdbf6cdc9fcd6c8 P 67def1cdfdb048899741175fab806933 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:08:53.079839 7802 ts_tablet_manager.cc:1905] T e7028b69a2ad4604acdbf6cdc9fcd6c8 P 67def1cdfdb048899741175fab806933: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 04:08:53.079869 7350 connection.cc:535] server connection from 127.0.0.1:54070 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260924 04:08:53.080183 7625 connection.cc:535] server connection from 127.0.0.1:45942 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 04:08:53.082564 7802 ts_tablet_manager.cc:1918] T e7028b69a2ad4604acdbf6cdc9fcd6c8 P 67def1cdfdb048899741175fab806933: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:08:53.082676 7802 log.cc:1196] T e7028b69a2ad4604acdbf6cdc9fcd6c8 P 67def1cdfdb048899741175fab806933: Deleting WAL directory at /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/ts-1/wal/wals/e7028b69a2ad4604acdbf6cdc9fcd6c8
I20260924 04:08:53.083041 7802 ts_tablet_manager.cc:1939] T e7028b69a2ad4604acdbf6cdc9fcd6c8 P 67def1cdfdb048899741175fab806933: Deleting consensus metadata
W20260924 04:08:53.083979 7486 connection.cc:441] server connection from 127.0.0.1:42982 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42982 (request call id 1) could send its response
I20260924 04:08:53.098259 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:53.071642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42252 (local address 127.0.98.61:43705)
0924 04:08:53.078113 (+ 6471us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:53.078117 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:53.080418 (+ 2301us) server_negotiation.cc:407] Connection header received
0924 04:08:53.080467 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:53.080470 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:53.080524 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:53.080597 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:53.081518 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:53.084387 (+ 2869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:53.085615 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:53.085972 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:53.086062 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:53.094006 (+ 7944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:53.094010 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:53.094014 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:53.094043 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:53.096047 (+ 2004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:53.096733 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:53.096736 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:53.096738 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:53.096825 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:53.097146 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:53.097149 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:53.097151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:53.097383 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:53.097491 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:53.097906 (+ 415us) server_negotiation.cc:299] Negotiation successful
0924 04:08:53.098079 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":6394,"thread_start_us":54,"threads_started":1}
I20260924 04:08:53.098404 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:53.084604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49660 (local address 127.0.98.60:34051)
0924 04:08:53.085121 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:53.085125 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:53.085436 (+ 311us) server_negotiation.cc:407] Connection header received
0924 04:08:53.085486 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:53.085489 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:53.085544 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:53.085631 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:53.086523 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:53.087267 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:53.088455 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:53.088785 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:53.088874 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:53.093629 (+ 4755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:53.093634 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:53.093638 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:53.093673 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:53.096069 (+ 2396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:53.096792 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:53.096796 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:53.096797 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:53.096889 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:53.097156 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:53.097159 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:53.097161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:53.097376 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:53.097492 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:53.098121 (+ 629us) server_negotiation.cc:299] Negotiation successful
0924 04:08:53.098230 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":81,"threads_started":1}
I20260924 04:08:53.098886 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:53.084107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42264 (local address 127.0.98.61:43705)
0924 04:08:53.084330 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:53.084334 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:53.084554 (+ 220us) server_negotiation.cc:407] Connection header received
0924 04:08:53.085001 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:53.085005 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:53.085054 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:53.085113 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:53.086121 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:53.086893 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:53.087839 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:53.088087 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:53.088202 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:53.093862 (+ 5660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:53.093866 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:53.093869 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:53.093893 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:53.096185 (+ 2292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:53.096817 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:53.096821 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:53.096823 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:53.096903 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:53.097207 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:53.097210 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:53.097212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:53.097416 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:53.097527 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:53.097868 (+ 341us) server_negotiation.cc:299] Negotiation successful
0924 04:08:53.098001 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":72,"threads_started":1}
I20260924 04:08:53.107950 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:53.071377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49644 (local address 127.0.98.60:34051)
0924 04:08:53.076868 (+ 5491us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:53.076874 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:53.099678 (+ 22804us) server_negotiation.cc:407] Connection header received
0924 04:08:53.099882 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:53.099886 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:53.099937 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:53.100001 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:53.100815 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:53.101605 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:53.102361 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:53.102708 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:53.102819 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:53.104160 (+ 1341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:53.104164 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:53.104166 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:53.104191 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:53.106381 (+ 2190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:53.106778 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:53.106783 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:53.106785 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:53.106864 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:53.107129 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:53.107133 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:53.107136 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:53.107300 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:53.107437 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:53.107671 (+ 234us) server_negotiation.cc:299] Negotiation successful
0924 04:08:53.107786 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5382,"thread_start_us":73,"threads_started":1}
I20260924 04:08:54.057006 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:54.056452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38949 (local address 127.0.98.1:33425)
0924 04:08:54.056764 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:54.056850 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":110,"threads_started":1}
W20260924 04:08:54.057636 7462 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38949 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:38949: connect: Connection refused (error 111)
I20260924 04:08:54.058233 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:54.057679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38949 (local address 127.0.98.3:50171)
0924 04:08:54.058010 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:54.058101 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":74,"threads_started":1}
W20260924 04:08:54.058490 7736 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38949 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:38949: connect: Connection refused (error 111)
I20260924 04:08:54.062827 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:54.062433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38949 (local address 127.0.98.2:40369)
0924 04:08:54.062674 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:54.062745 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":68,"threads_started":1}
W20260924 04:08:54.063149 7599 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38949 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:38949: connect: Connection refused (error 111)
I20260924 04:08:54.568209 7819 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c01afc87f8b4560b973c517eea256a2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cd568f5711864cc4a75a0f2fef7236a2)
I20260924 04:08:54.568430 7819 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c01afc87f8b4560b973c517eea256a2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cd568f5711864cc4a75a0f2fef7236a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38949 } } peers { permanent_uuid: "3c01afc87f8b4560b973c517eea256a2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43705 } attrs { promote: false } } peers { permanent_uuid: "90d32bb2c3d041d78f61550aa83a8a15" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34051 } attrs { promote: false } }
I20260924 04:08:54.568758 7820 raft_consensus.cc:482] T 00000000000000000000000000000000 P 90d32bb2c3d041d78f61550aa83a8a15 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cd568f5711864cc4a75a0f2fef7236a2)
I20260924 04:08:54.568975 7820 raft_consensus.cc:504] T 00000000000000000000000000000000 P 90d32bb2c3d041d78f61550aa83a8a15 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cd568f5711864cc4a75a0f2fef7236a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38949 } } peers { permanent_uuid: "3c01afc87f8b4560b973c517eea256a2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43705 } attrs { promote: false } } peers { permanent_uuid: "90d32bb2c3d041d78f61550aa83a8a15" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34051 } attrs { promote: false } }
I20260924 04:08:54.569582 7819 leader_election.cc:288] T 00000000000000000000000000000000 P 3c01afc87f8b4560b973c517eea256a2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cd568f5711864cc4a75a0f2fef7236a2 (127.0.98.62:38949), 90d32bb2c3d041d78f61550aa83a8a15 (127.0.98.60:34051)
I20260924 04:08:54.570223 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:54.569756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38949 (local address 127.0.0.1:54128)
0924 04:08:54.570042 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:54.570119 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":58,"threads_started":1}
W20260924 04:08:54.570395 7083 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38949: connect: Connection refused (error 111)
I20260924 04:08:54.570526 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:54.569985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38949 (local address 127.0.0.1:54144)
0924 04:08:54.570258 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:54.570385 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":65,"threads_started":1}
I20260924 04:08:54.570673 7820 leader_election.cc:288] T 00000000000000000000000000000000 P 90d32bb2c3d041d78f61550aa83a8a15 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cd568f5711864cc4a75a0f2fef7236a2 (127.0.98.62:38949), 3c01afc87f8b4560b973c517eea256a2 (127.0.98.61:43705)
W20260924 04:08:54.570772 7266 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38949: connect: Connection refused (error 111)
I20260924 04:08:54.571656 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:54.571424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38949 (local address 127.0.0.1:54146)
0924 04:08:54.571534 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:54.571589 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260924 04:08:54.571753 7266 leader_election.cc:334] T 00000000000000000000000000000000 P 90d32bb2c3d041d78f61550aa83a8a15 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer cd568f5711864cc4a75a0f2fef7236a2 (127.0.98.62:38949): Network error: Client connection negotiation failed: client connection to 127.0.98.62:38949: connect: Connection refused (error 111)
I20260924 04:08:54.572062 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:54.571707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38949 (local address 127.0.0.1:54162)
0924 04:08:54.571938 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:54.571992 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155}
W20260924 04:08:54.572227 7083 leader_election.cc:334] T 00000000000000000000000000000000 P 3c01afc87f8b4560b973c517eea256a2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer cd568f5711864cc4a75a0f2fef7236a2 (127.0.98.62:38949): Network error: Client connection negotiation failed: client connection to 127.0.98.62:38949: connect: Connection refused (error 111)
I20260924 04:08:54.574332 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:54.569797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34051 (local address 127.0.0.1:49670)
0924 04:08:54.570096 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:54.570112 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:54.570218 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:54.570735 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:54.570738 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:54.570755 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:08:54.570873 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:54.570879 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:54.572018 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:54.572021 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:54.573029 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:54.573038 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:54.573867 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:54.573870 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:54.573959 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:54.573984 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 04:08:54.574067 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 04:08:54.574135 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":72,"threads_started":1}
I20260924 04:08:54.574339 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:54.569985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49670 (local address 127.0.98.60:34051)
0924 04:08:54.570471 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:54.570475 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:54.570490 (+ 15us) server_negotiation.cc:407] Connection header received
0924 04:08:54.570538 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:54.570544 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:54.570596 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:54.570983 (+ 387us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:08:54.571086 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:54.571866 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:54.573155 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:54.573749 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:54.573833 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:54.573878 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:54.574101 (+ 223us) server_negotiation.cc:299] Negotiation successful
0924 04:08:54.574207 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":51,"threads_started":1}
I20260924 04:08:54.574807 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:54.570724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42280 (local address 127.0.98.61:43705)
0924 04:08:54.571041 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:54.571052 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:54.571066 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:08:54.571110 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:54.571113 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:54.571153 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:54.571512 (+ 359us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:08:54.571608 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:54.572368 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:54.573615 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:54.574194 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:54.574292 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:54.574345 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:54.574676 (+ 331us) server_negotiation.cc:299] Negotiation successful
0924 04:08:54.574731 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":250,"thread_start_us":86,"threads_started":1}
I20260924 04:08:54.574772 7286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3c01afc87f8b4560b973c517eea256a2" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "90d32bb2c3d041d78f61550aa83a8a15" is_pre_election: true
I20260924 04:08:54.574870 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:54.570464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43705 (local address 127.0.0.1:42280)
0924 04:08:54.570874 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:54.570888 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:54.570976 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:54.571273 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:54.571276 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:54.571293 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:08:54.571396 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:54.571401 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:54.572477 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:54.572481 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:54.573487 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:54.573496 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:54.574301 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:54.574305 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:54.574398 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:54.574570 (+ 172us) client_negotiation.cc:769] Sending connection context
0924 04:08:54.574640 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 04:08:54.574723 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":61,"threads_started":1}
I20260924 04:08:54.574991 7286 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 90d32bb2c3d041d78f61550aa83a8a15 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c01afc87f8b4560b973c517eea256a2 in term 1.
I20260924 04:08:54.575243 7109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "90d32bb2c3d041d78f61550aa83a8a15" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "3c01afc87f8b4560b973c517eea256a2" is_pre_election: true
I20260924 04:08:54.575350 7084 leader_election.cc:302] T 00000000000000000000000000000000 P 3c01afc87f8b4560b973c517eea256a2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c01afc87f8b4560b973c517eea256a2, 90d32bb2c3d041d78f61550aa83a8a15; no voters: cd568f5711864cc4a75a0f2fef7236a2
I20260924 04:08:54.575419 7109 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3c01afc87f8b4560b973c517eea256a2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90d32bb2c3d041d78f61550aa83a8a15 in term 1.
I20260924 04:08:54.575546 7819 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c01afc87f8b4560b973c517eea256a2 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 04:08:54.575613 7819 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c01afc87f8b4560b973c517eea256a2 [term 1 FOLLOWER]: Starting leader election (detected failure of leader cd568f5711864cc4a75a0f2fef7236a2)
I20260924 04:08:54.575659 7819 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c01afc87f8b4560b973c517eea256a2 [term 1 FOLLOWER]: Advancing to term 2
I20260924 04:08:54.575707 7267 leader_election.cc:302] T 00000000000000000000000000000000 P 90d32bb2c3d041d78f61550aa83a8a15 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c01afc87f8b4560b973c517eea256a2, 90d32bb2c3d041d78f61550aa83a8a15; no voters: cd568f5711864cc4a75a0f2fef7236a2
I20260924 04:08:54.575884 7820 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 90d32bb2c3d041d78f61550aa83a8a15 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 04:08:54.575930 7820 raft_consensus.cc:482] T 00000000000000000000000000000000 P 90d32bb2c3d041d78f61550aa83a8a15 [term 1 FOLLOWER]: Starting leader election (detected failure of leader cd568f5711864cc4a75a0f2fef7236a2)
I20260924 04:08:54.575968 7820 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 90d32bb2c3d041d78f61550aa83a8a15 [term 1 FOLLOWER]: Advancing to term 2
I20260924 04:08:54.576629 7819 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c01afc87f8b4560b973c517eea256a2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cd568f5711864cc4a75a0f2fef7236a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38949 } } peers { permanent_uuid: "3c01afc87f8b4560b973c517eea256a2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43705 } attrs { promote: false } } peers { permanent_uuid: "90d32bb2c3d041d78f61550aa83a8a15" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34051 } attrs { promote: false } }
I20260924 04:08:54.576938 7820 raft_consensus.cc:504] T 00000000000000000000000000000000 P 90d32bb2c3d041d78f61550aa83a8a15 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cd568f5711864cc4a75a0f2fef7236a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38949 } } peers { permanent_uuid: "3c01afc87f8b4560b973c517eea256a2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43705 } attrs { promote: false } } peers { permanent_uuid: "90d32bb2c3d041d78f61550aa83a8a15" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34051 } attrs { promote: false } }
I20260924 04:08:54.577040 7819 leader_election.cc:288] T 00000000000000000000000000000000 P 3c01afc87f8b4560b973c517eea256a2 [CANDIDATE]: Term 2 election: Requested vote from peers cd568f5711864cc4a75a0f2fef7236a2 (127.0.98.62:38949), 90d32bb2c3d041d78f61550aa83a8a15 (127.0.98.60:34051)
I20260924 04:08:54.577329 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:54.577112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38949 (local address 127.0.0.1:54164)
0924 04:08:54.577216 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:54.577270 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260924 04:08:54.577332 7820 leader_election.cc:288] T 00000000000000000000000000000000 P 90d32bb2c3d041d78f61550aa83a8a15 [CANDIDATE]: Term 2 election: Requested vote from peers cd568f5711864cc4a75a0f2fef7236a2 (127.0.98.62:38949), 3c01afc87f8b4560b973c517eea256a2 (127.0.98.61:43705)
I20260924 04:08:54.577670 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:54.577440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38949 (local address 127.0.0.1:54172)
0924 04:08:54.577553 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:54.577600 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260924 04:08:54.577693 7286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3c01afc87f8b4560b973c517eea256a2" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "90d32bb2c3d041d78f61550aa83a8a15"
I20260924 04:08:54.577842 7286 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 90d32bb2c3d041d78f61550aa83a8a15 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 3c01afc87f8b4560b973c517eea256a2 in current term 2: Already voted for candidate 90d32bb2c3d041d78f61550aa83a8a15 in this term.
I20260924 04:08:54.577924 7109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "90d32bb2c3d041d78f61550aa83a8a15" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "3c01afc87f8b4560b973c517eea256a2"
I20260924 04:08:54.578073 7109 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 3c01afc87f8b4560b973c517eea256a2 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 90d32bb2c3d041d78f61550aa83a8a15 in current term 2: Already voted for candidate 3c01afc87f8b4560b973c517eea256a2 in this term.
I20260924 04:08:54.578174 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:54.577803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38949 (local address 127.0.0.1:54178)
0924 04:08:54.578058 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:54.578103 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197}
W20260924 04:08:54.578272 7083 leader_election.cc:334] T 00000000000000000000000000000000 P 3c01afc87f8b4560b973c517eea256a2 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer cd568f5711864cc4a75a0f2fef7236a2 (127.0.98.62:38949): Network error: Client connection negotiation failed: client connection to 127.0.98.62:38949: connect: Connection refused (error 111)
I20260924 04:08:54.578347 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:54.578052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38949 (local address 127.0.0.1:54192)
0924 04:08:54.578246 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:54.578297 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
I20260924 04:08:54.578338 7083 leader_election.cc:302] T 00000000000000000000000000000000 P 3c01afc87f8b4560b973c517eea256a2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c01afc87f8b4560b973c517eea256a2; no voters: 90d32bb2c3d041d78f61550aa83a8a15, cd568f5711864cc4a75a0f2fef7236a2
W20260924 04:08:54.578421 7266 leader_election.cc:334] T 00000000000000000000000000000000 P 90d32bb2c3d041d78f61550aa83a8a15 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer cd568f5711864cc4a75a0f2fef7236a2 (127.0.98.62:38949): Network error: Client connection negotiation failed: client connection to 127.0.98.62:38949: connect: Connection refused (error 111)
I20260924 04:08:54.578465 7819 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 3c01afc87f8b4560b973c517eea256a2 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260924 04:08:54.578478 7266 leader_election.cc:302] T 00000000000000000000000000000000 P 90d32bb2c3d041d78f61550aa83a8a15 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90d32bb2c3d041d78f61550aa83a8a15; no voters: 3c01afc87f8b4560b973c517eea256a2, cd568f5711864cc4a75a0f2fef7236a2
I20260924 04:08:54.578606 7820 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 90d32bb2c3d041d78f61550aa83a8a15 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260924 04:08:55.058808 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:55.058303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38949 (local address 127.0.98.1:58093)
0924 04:08:55.058600 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:55.058697 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":104,"threads_started":1}
I20260924 04:08:55.059868 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:55.059417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38949 (local address 127.0.98.3:33619)
0924 04:08:55.059721 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:55.059777 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":98,"threads_started":1}
I20260924 04:08:55.064172 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:55.063787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38949 (local address 127.0.98.2:55877)
0924 04:08:55.064037 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:55.064091 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":83,"threads_started":1}
I20260924 04:08:56.060565 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:56.059828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38949 (local address 127.0.98.1:36145)
0924 04:08:56.060142 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:56.060204 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":128,"threads_started":1}
I20260924 04:08:56.061317 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:56.060498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38949 (local address 127.0.98.3:52229)
0924 04:08:56.060971 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:56.061030 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":75,"threads_started":1}
I20260924 04:08:56.065224 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:56.064870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38949 (local address 127.0.98.2:35609)
0924 04:08:56.065098 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:56.065150 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":81,"threads_started":1}
I20260924 04:08:56.170449 7835 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c01afc87f8b4560b973c517eea256a2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:08:56.170570 7835 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c01afc87f8b4560b973c517eea256a2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cd568f5711864cc4a75a0f2fef7236a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38949 } } peers { permanent_uuid: "3c01afc87f8b4560b973c517eea256a2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43705 } attrs { promote: false } } peers { permanent_uuid: "90d32bb2c3d041d78f61550aa83a8a15" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34051 } attrs { promote: false } }
I20260924 04:08:56.171058 7835 leader_election.cc:288] T 00000000000000000000000000000000 P 3c01afc87f8b4560b973c517eea256a2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd568f5711864cc4a75a0f2fef7236a2 (127.0.98.62:38949), 90d32bb2c3d041d78f61550aa83a8a15 (127.0.98.60:34051)
I20260924 04:08:56.171530 7286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3c01afc87f8b4560b973c517eea256a2" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "90d32bb2c3d041d78f61550aa83a8a15" is_pre_election: true
I20260924 04:08:56.171685 7286 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 90d32bb2c3d041d78f61550aa83a8a15 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c01afc87f8b4560b973c517eea256a2 in term 2.
I20260924 04:08:56.171726 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:56.171307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38949 (local address 127.0.0.1:54416)
0924 04:08:56.171539 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:56.171602 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":94,"threads_started":1}
I20260924 04:08:56.172043 7084 leader_election.cc:302] T 00000000000000000000000000000000 P 3c01afc87f8b4560b973c517eea256a2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c01afc87f8b4560b973c517eea256a2, 90d32bb2c3d041d78f61550aa83a8a15; no voters:
I20260924 04:08:56.172227 7835 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c01afc87f8b4560b973c517eea256a2 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260924 04:08:56.172284 7835 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c01afc87f8b4560b973c517eea256a2 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:08:56.172346 7835 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c01afc87f8b4560b973c517eea256a2 [term 2 FOLLOWER]: Advancing to term 3
I20260924 04:08:56.172534 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:56.172347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38949 (local address 127.0.0.1:54422)
0924 04:08:56.172436 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:56.172486 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260924 04:08:56.172632 7083 leader_election.cc:334] T 00000000000000000000000000000000 P 3c01afc87f8b4560b973c517eea256a2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer cd568f5711864cc4a75a0f2fef7236a2 (127.0.98.62:38949): Network error: Client connection negotiation failed: client connection to 127.0.98.62:38949: connect: Connection refused (error 111)
I20260924 04:08:56.173471 7835 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c01afc87f8b4560b973c517eea256a2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cd568f5711864cc4a75a0f2fef7236a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38949 } } peers { permanent_uuid: "3c01afc87f8b4560b973c517eea256a2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43705 } attrs { promote: false } } peers { permanent_uuid: "90d32bb2c3d041d78f61550aa83a8a15" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34051 } attrs { promote: false } }
I20260924 04:08:56.173821 7835 leader_election.cc:288] T 00000000000000000000000000000000 P 3c01afc87f8b4560b973c517eea256a2 [CANDIDATE]: Term 3 election: Requested vote from peers cd568f5711864cc4a75a0f2fef7236a2 (127.0.98.62:38949), 90d32bb2c3d041d78f61550aa83a8a15 (127.0.98.60:34051)
I20260924 04:08:56.174168 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:56.173935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38949 (local address 127.0.0.1:54432)
0924 04:08:56.174023 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:56.174075 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260924 04:08:56.174166 7286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3c01afc87f8b4560b973c517eea256a2" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "90d32bb2c3d041d78f61550aa83a8a15"
I20260924 04:08:56.174333 7286 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 90d32bb2c3d041d78f61550aa83a8a15 [term 2 FOLLOWER]: Advancing to term 3
I20260924 04:08:56.174786 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:56.174587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38949 (local address 127.0.0.1:54436)
0924 04:08:56.174688 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:56.174734 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260924 04:08:56.174921 7083 leader_election.cc:334] T 00000000000000000000000000000000 P 3c01afc87f8b4560b973c517eea256a2 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer cd568f5711864cc4a75a0f2fef7236a2 (127.0.98.62:38949): Network error: Client connection negotiation failed: client connection to 127.0.98.62:38949: connect: Connection refused (error 111)
I20260924 04:08:56.175403 7286 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 90d32bb2c3d041d78f61550aa83a8a15 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c01afc87f8b4560b973c517eea256a2 in term 3.
I20260924 04:08:56.175786 7084 leader_election.cc:302] T 00000000000000000000000000000000 P 3c01afc87f8b4560b973c517eea256a2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c01afc87f8b4560b973c517eea256a2, 90d32bb2c3d041d78f61550aa83a8a15; no voters: cd568f5711864cc4a75a0f2fef7236a2
I20260924 04:08:56.175947 7835 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c01afc87f8b4560b973c517eea256a2 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 04:08:56.176119 7835 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c01afc87f8b4560b973c517eea256a2 [term 3 LEADER]: Becoming Leader. State: Replica: 3c01afc87f8b4560b973c517eea256a2, State: Running, Role: LEADER
I20260924 04:08:56.176393 7835 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c01afc87f8b4560b973c517eea256a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cd568f5711864cc4a75a0f2fef7236a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38949 } } peers { permanent_uuid: "3c01afc87f8b4560b973c517eea256a2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43705 } attrs { promote: false } } peers { permanent_uuid: "90d32bb2c3d041d78f61550aa83a8a15" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34051 } attrs { promote: false } }
I20260924 04:08:56.177886 7838 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c01afc87f8b4560b973c517eea256a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c01afc87f8b4560b973c517eea256a2. Latest consensus state: current_term: 3 leader_uuid: "3c01afc87f8b4560b973c517eea256a2" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cd568f5711864cc4a75a0f2fef7236a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38949 } } peers { permanent_uuid: "3c01afc87f8b4560b973c517eea256a2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43705 } attrs { promote: false } } peers { permanent_uuid: "90d32bb2c3d041d78f61550aa83a8a15" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34051 } attrs { promote: false } } }
I20260924 04:08:56.178018 7838 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c01afc87f8b4560b973c517eea256a2 [sys.catalog]: This master's current role is: LEADER
I20260924 04:08:56.178309 7840 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:08:56.180524 7840 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:08:56.180953 7840 catalog_manager.cc:1259] Loaded cluster ID: 66060a64b3b6445ca22111ebb637e30d
I20260924 04:08:56.180999 7840 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:08:56.181495 7840 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:08:56.181867 7840 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3c01afc87f8b4560b973c517eea256a2: Loaded TSK: 0
I20260924 04:08:56.182555 7840 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:08:56.230116 7091 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42264:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:08:56.233167 7091 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:08:56.236064 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:56.235805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38949 (local address 127.0.0.1:54448)
0924 04:08:56.235936 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:56.235994 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260924 04:08:56.236100 7286 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 90d32bb2c3d041d78f61550aa83a8a15 [term 3 FOLLOWER]: Refusing update from remote peer 3c01afc87f8b4560b973c517eea256a2: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260924 04:08:56.236632 7835 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3c01afc87f8b4560b973c517eea256a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d32bb2c3d041d78f61550aa83a8a15" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34051 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260924 04:08:56.237206 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:56.236966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38949 (local address 127.0.0.1:54456)
0924 04:08:56.237084 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:56.237141 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260924 04:08:56.237536 7083 consensus_peers.cc:487] T 00000000000000000000000000000000 P 3c01afc87f8b4560b973c517eea256a2 -> Peer cd568f5711864cc4a75a0f2fef7236a2 (127.0.98.62:38949): Couldn't send request to peer cd568f5711864cc4a75a0f2fef7236a2. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38949: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 04:08:56.238446 7842 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90d32bb2c3d041d78f61550aa83a8a15 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c01afc87f8b4560b973c517eea256a2. Latest consensus state: current_term: 3 leader_uuid: "3c01afc87f8b4560b973c517eea256a2" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cd568f5711864cc4a75a0f2fef7236a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38949 } } peers { permanent_uuid: "3c01afc87f8b4560b973c517eea256a2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43705 } attrs { promote: false } } peers { permanent_uuid: "90d32bb2c3d041d78f61550aa83a8a15" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34051 } attrs { promote: false } } }
I20260924 04:08:56.238598 7842 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90d32bb2c3d041d78f61550aa83a8a15 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:08:56.240011 7838 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c01afc87f8b4560b973c517eea256a2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "3c01afc87f8b4560b973c517eea256a2" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cd568f5711864cc4a75a0f2fef7236a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38949 } } peers { permanent_uuid: "3c01afc87f8b4560b973c517eea256a2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43705 } attrs { promote: false } } peers { permanent_uuid: "90d32bb2c3d041d78f61550aa83a8a15" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34051 } attrs { promote: false } } }
I20260924 04:08:56.240141 7838 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c01afc87f8b4560b973c517eea256a2 [sys.catalog]: This master's current role is: LEADER
I20260924 04:08:56.240326 7842 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90d32bb2c3d041d78f61550aa83a8a15 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "3c01afc87f8b4560b973c517eea256a2" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cd568f5711864cc4a75a0f2fef7236a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38949 } } peers { permanent_uuid: "3c01afc87f8b4560b973c517eea256a2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43705 } attrs { promote: false } } peers { permanent_uuid: "90d32bb2c3d041d78f61550aa83a8a15" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34051 } attrs { promote: false } } }
I20260924 04:08:56.240427 7842 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90d32bb2c3d041d78f61550aa83a8a15 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:08:56.250197 7847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:56.246129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45954 (local address 127.0.98.3:33301)
0924 04:08:56.246521 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:56.246526 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:56.246540 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:08:56.246591 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:56.246595 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:56.246653 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:56.247007 (+ 354us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:08:56.247116 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:56.247852 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:56.249014 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:56.249625 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:56.249741 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:56.249820 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:56.249983 (+ 163us) server_negotiation.cc:299] Negotiation successful
0924 04:08:56.250058 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":71,"threads_started":1}
I20260924 04:08:56.250176 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:56.245984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33301 (local address 127.0.0.1:45954)
0924 04:08:56.246101 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:56.246118 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:56.246204 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:56.246772 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:56.246774 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:56.246787 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:08:56.246896 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:56.246901 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:56.247954 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:56.247959 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:56.248887 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:56.248896 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:56.249744 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:56.249748 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:56.249851 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:56.249891 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 04:08:56.249963 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 04:08:56.250029 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":3}
I20260924 04:08:56.251047 7848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:56.246199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37085 (local address 127.0.0.1:43010)
0924 04:08:56.246578 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:56.246591 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:56.246664 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:56.247513 (+ 849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:56.247516 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:56.247531 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:08:56.247612 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:56.247616 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:56.248554 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:56.248557 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:56.249824 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:56.249832 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:56.250665 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:56.250669 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:56.250763 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:56.250793 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 04:08:56.250866 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 04:08:56.250929 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"mutex_wait_us":128,"thread_start_us":105,"threads_started":1}
I20260924 04:08:56.251070 7669 tablet_service.cc:1467] Processing CreateTablet for tablet 7c75237d6b9e4f22a57b5afee30951f7 (DEFAULT_TABLE table=test-table [id=08a650465565473dac65e340ecdccd2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:08:56.251379 7669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c75237d6b9e4f22a57b5afee30951f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:08:56.251494 7850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:56.246528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43010 (local address 127.0.98.2:37085)
0924 04:08:56.247269 (+ 741us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:56.247274 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:56.247283 (+ 9us) server_negotiation.cc:407] Connection header received
0924 04:08:56.247322 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:56.247324 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:56.247385 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:56.247495 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:08:56.247709 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:56.248445 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:56.249920 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:56.250552 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:56.250669 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:56.250726 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:56.251283 (+ 557us) server_negotiation.cc:299] Negotiation successful
0924 04:08:56.251371 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":673,"thread_start_us":82,"threads_started":1}
I20260924 04:08:56.251840 7849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:56.246475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44685 (local address 127.0.0.1:54080)
0924 04:08:56.247050 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:56.247062 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:56.247137 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:56.247404 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:56.247410 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:56.247430 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:08:56.247537 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:56.247543 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:56.248560 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:56.248563 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:56.250207 (+ 1644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:56.250215 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:56.251409 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:56.251411 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:56.251522 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:56.251553 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 04:08:56.251625 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 04:08:56.251698 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"mutex_wait_us":62,"thread_start_us":77,"threads_started":1}
I20260924 04:08:56.251971 7532 tablet_service.cc:1467] Processing CreateTablet for tablet 7c75237d6b9e4f22a57b5afee30951f7 (DEFAULT_TABLE table=test-table [id=08a650465565473dac65e340ecdccd2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:08:56.252221 7851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:56.246883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54080 (local address 127.0.98.1:44685)
0924 04:08:56.247184 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:56.247188 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:56.247201 (+ 13us) server_negotiation.cc:407] Connection header received
0924 04:08:56.247248 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:56.247251 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:56.247293 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:56.247384 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:08:56.247633 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:56.248426 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:56.250446 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:56.251180 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:56.251288 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:56.251350 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:56.251991 (+ 641us) server_negotiation.cc:299] Negotiation successful
0924 04:08:56.252077 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":73,"threads_started":1}
I20260924 04:08:56.252231 7532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c75237d6b9e4f22a57b5afee30951f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:08:56.252739 7395 tablet_service.cc:1467] Processing CreateTablet for tablet 7c75237d6b9e4f22a57b5afee30951f7 (DEFAULT_TABLE table=test-table [id=08a650465565473dac65e340ecdccd2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:08:56.253017 7395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c75237d6b9e4f22a57b5afee30951f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:08:56.254103 7852 tablet_bootstrap.cc:492] T 7c75237d6b9e4f22a57b5afee30951f7 P a5a609a5eb2a40cfaf3f6616900bd505: Bootstrap starting.
I20260924 04:08:56.254587 7853 tablet_bootstrap.cc:492] T 7c75237d6b9e4f22a57b5afee30951f7 P 67def1cdfdb048899741175fab806933: Bootstrap starting.
I20260924 04:08:56.255017 7852 tablet_bootstrap.cc:654] T 7c75237d6b9e4f22a57b5afee30951f7 P a5a609a5eb2a40cfaf3f6616900bd505: Neither blocks nor log segments found. Creating new log.
I20260924 04:08:56.255436 7853 tablet_bootstrap.cc:654] T 7c75237d6b9e4f22a57b5afee30951f7 P 67def1cdfdb048899741175fab806933: Neither blocks nor log segments found. Creating new log.
I20260924 04:08:56.256443 7852 tablet_bootstrap.cc:492] T 7c75237d6b9e4f22a57b5afee30951f7 P a5a609a5eb2a40cfaf3f6616900bd505: No bootstrap required, opened a new log
I20260924 04:08:56.256613 7854 tablet_bootstrap.cc:492] T 7c75237d6b9e4f22a57b5afee30951f7 P dc6019f5f5474832b90b9150dcc4b0b3: Bootstrap starting.
I20260924 04:08:56.256664 7853 tablet_bootstrap.cc:492] T 7c75237d6b9e4f22a57b5afee30951f7 P 67def1cdfdb048899741175fab806933: No bootstrap required, opened a new log
I20260924 04:08:56.256664 7852 ts_tablet_manager.cc:1397] T 7c75237d6b9e4f22a57b5afee30951f7 P a5a609a5eb2a40cfaf3f6616900bd505: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 04:08:56.256871 7853 ts_tablet_manager.cc:1397] T 7c75237d6b9e4f22a57b5afee30951f7 P 67def1cdfdb048899741175fab806933: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260924 04:08:56.257165 7852 raft_consensus.cc:348] T 7c75237d6b9e4f22a57b5afee30951f7 P a5a609a5eb2a40cfaf3f6616900bd505 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5a609a5eb2a40cfaf3f6616900bd505" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33301 } } peers { permanent_uuid: "67def1cdfdb048899741175fab806933" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37085 } } peers { permanent_uuid: "dc6019f5f5474832b90b9150dcc4b0b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44685 } }
I20260924 04:08:56.257292 7852 raft_consensus.cc:374] T 7c75237d6b9e4f22a57b5afee30951f7 P a5a609a5eb2a40cfaf3f6616900bd505 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:08:56.257349 7852 raft_consensus.cc:729] T 7c75237d6b9e4f22a57b5afee30951f7 P a5a609a5eb2a40cfaf3f6616900bd505 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5a609a5eb2a40cfaf3f6616900bd505, State: Initialized, Role: FOLLOWER
I20260924 04:08:56.257301 7853 raft_consensus.cc:348] T 7c75237d6b9e4f22a57b5afee30951f7 P 67def1cdfdb048899741175fab806933 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5a609a5eb2a40cfaf3f6616900bd505" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33301 } } peers { permanent_uuid: "67def1cdfdb048899741175fab806933" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37085 } } peers { permanent_uuid: "dc6019f5f5474832b90b9150dcc4b0b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44685 } }
I20260924 04:08:56.257407 7853 raft_consensus.cc:374] T 7c75237d6b9e4f22a57b5afee30951f7 P 67def1cdfdb048899741175fab806933 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:08:56.257494 7853 raft_consensus.cc:729] T 7c75237d6b9e4f22a57b5afee30951f7 P 67def1cdfdb048899741175fab806933 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67def1cdfdb048899741175fab806933, State: Initialized, Role: FOLLOWER
I20260924 04:08:56.257534 7854 tablet_bootstrap.cc:654] T 7c75237d6b9e4f22a57b5afee30951f7 P dc6019f5f5474832b90b9150dcc4b0b3: Neither blocks nor log segments found. Creating new log.
I20260924 04:08:56.257534 7852 consensus_queue.cc:260] T 7c75237d6b9e4f22a57b5afee30951f7 P a5a609a5eb2a40cfaf3f6616900bd505 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5a609a5eb2a40cfaf3f6616900bd505" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33301 } } peers { permanent_uuid: "67def1cdfdb048899741175fab806933" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37085 } } peers { permanent_uuid: "dc6019f5f5474832b90b9150dcc4b0b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44685 } }
I20260924 04:08:56.257669 7853 consensus_queue.cc:260] T 7c75237d6b9e4f22a57b5afee30951f7 P 67def1cdfdb048899741175fab806933 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5a609a5eb2a40cfaf3f6616900bd505" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33301 } } peers { permanent_uuid: "67def1cdfdb048899741175fab806933" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37085 } } peers { permanent_uuid: "dc6019f5f5474832b90b9150dcc4b0b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44685 } }
I20260924 04:08:56.258026 7853 ts_tablet_manager.cc:1428] T 7c75237d6b9e4f22a57b5afee30951f7 P 67def1cdfdb048899741175fab806933: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260924 04:08:56.258411 7852 ts_tablet_manager.cc:1428] T 7c75237d6b9e4f22a57b5afee30951f7 P a5a609a5eb2a40cfaf3f6616900bd505: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260924 04:08:56.258563 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:56.258274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38949 (local address 127.0.98.3:48873)
0924 04:08:56.258431 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:56.258493 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260924 04:08:56.259558 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:56.259003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38949 (local address 127.0.98.3:41393)
0924 04:08:56.259426 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:56.259475 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358}
I20260924 04:08:56.259829 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:56.259376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38949 (local address 127.0.98.2:41721)
0924 04:08:56.259724 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:56.259777 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":274}
I20260924 04:08:56.260594 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:56.260306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38949 (local address 127.0.98.2:60193)
0924 04:08:56.260486 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:56.260535 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260924 04:08:56.261482 7737 heartbeater.cc:503] Master 127.0.98.61:43705 was elected leader, sending a full tablet report...
I20260924 04:08:56.261596 7600 heartbeater.cc:503] Master 127.0.98.61:43705 was elected leader, sending a full tablet report...
I20260924 04:08:56.262555 7854 tablet_bootstrap.cc:492] T 7c75237d6b9e4f22a57b5afee30951f7 P dc6019f5f5474832b90b9150dcc4b0b3: No bootstrap required, opened a new log
I20260924 04:08:56.262686 7854 ts_tablet_manager.cc:1397] T 7c75237d6b9e4f22a57b5afee30951f7 P dc6019f5f5474832b90b9150dcc4b0b3: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260924 04:08:56.263111 7854 raft_consensus.cc:348] T 7c75237d6b9e4f22a57b5afee30951f7 P dc6019f5f5474832b90b9150dcc4b0b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5a609a5eb2a40cfaf3f6616900bd505" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33301 } } peers { permanent_uuid: "67def1cdfdb048899741175fab806933" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37085 } } peers { permanent_uuid: "dc6019f5f5474832b90b9150dcc4b0b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44685 } }
I20260924 04:08:56.263206 7854 raft_consensus.cc:374] T 7c75237d6b9e4f22a57b5afee30951f7 P dc6019f5f5474832b90b9150dcc4b0b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:08:56.263227 7854 raft_consensus.cc:729] T 7c75237d6b9e4f22a57b5afee30951f7 P dc6019f5f5474832b90b9150dcc4b0b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc6019f5f5474832b90b9150dcc4b0b3, State: Initialized, Role: FOLLOWER
I20260924 04:08:56.263356 7854 consensus_queue.cc:260] T 7c75237d6b9e4f22a57b5afee30951f7 P dc6019f5f5474832b90b9150dcc4b0b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5a609a5eb2a40cfaf3f6616900bd505" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33301 } } peers { permanent_uuid: "67def1cdfdb048899741175fab806933" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37085 } } peers { permanent_uuid: "dc6019f5f5474832b90b9150dcc4b0b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44685 } }
I20260924 04:08:56.263741 7854 ts_tablet_manager.cc:1428] T 7c75237d6b9e4f22a57b5afee30951f7 P dc6019f5f5474832b90b9150dcc4b0b3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:08:56.264355 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:56.264042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38949 (local address 127.0.98.1:36609)
0924 04:08:56.264219 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:56.264281 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260924 04:08:56.265204 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:56.264914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38949 (local address 127.0.98.1:43049)
0924 04:08:56.265085 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:56.265145 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260924 04:08:56.266064 7463 heartbeater.cc:503] Master 127.0.98.61:43705 was elected leader, sending a full tablet report...
I20260924 04:08:56.474251 7860 raft_consensus.cc:482] T 7c75237d6b9e4f22a57b5afee30951f7 P dc6019f5f5474832b90b9150dcc4b0b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:08:56.474406 7860 raft_consensus.cc:504] T 7c75237d6b9e4f22a57b5afee30951f7 P dc6019f5f5474832b90b9150dcc4b0b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5a609a5eb2a40cfaf3f6616900bd505" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33301 } } peers { permanent_uuid: "67def1cdfdb048899741175fab806933" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37085 } } peers { permanent_uuid: "dc6019f5f5474832b90b9150dcc4b0b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44685 } }
I20260924 04:08:56.475535 7860 leader_election.cc:288] T 7c75237d6b9e4f22a57b5afee30951f7 P dc6019f5f5474832b90b9150dcc4b0b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5a609a5eb2a40cfaf3f6616900bd505 (127.0.98.3:33301), 67def1cdfdb048899741175fab806933 (127.0.98.2:37085)
I20260924 04:08:56.479652 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:56.475929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33301 (local address 127.0.98.1:41951)
0924 04:08:56.476055 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:56.476073 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:56.476186 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:56.476541 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:56.476544 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:56.476571 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:08:56.476734 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:56.476747 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:56.477624 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:56.477627 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:56.478515 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:56.478524 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:56.479208 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:56.479212 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:56.479338 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:56.479364 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 04:08:56.479437 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 04:08:56.479504 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260924 04:08:56.479683 7847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:56.476031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41951 (local address 127.0.98.3:33301)
0924 04:08:56.476227 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:56.476233 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:56.476257 (+ 24us) server_negotiation.cc:407] Connection header received
0924 04:08:56.476318 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:56.476321 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:56.476375 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:56.476471 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:08:56.476855 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:56.477516 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:56.478647 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:56.479077 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:56.479154 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:56.479200 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:56.479466 (+ 266us) server_negotiation.cc:299] Negotiation successful
0924 04:08:56.479534 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
I20260924 04:08:56.479799 7850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:56.476351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45491 (local address 127.0.98.2:37085)
0924 04:08:56.476493 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:56.476500 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:56.476665 (+ 165us) server_negotiation.cc:407] Connection header received
0924 04:08:56.476801 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:56.476806 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:56.476869 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:56.476944 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:08:56.477213 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:56.477787 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:56.478792 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:56.479228 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:56.479338 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:56.479398 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:56.479603 (+ 205us) server_negotiation.cc:299] Negotiation successful
0924 04:08:56.479681 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 04:08:56.479851 7861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:56.476217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37085 (local address 127.0.98.1:45491)
0924 04:08:56.476565 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:56.476579 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:56.476673 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:56.476974 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:56.476977 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:56.476995 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:08:56.477111 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:56.477118 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:56.477894 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:56.477897 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:56.478677 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:56.478686 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:56.479360 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:56.479363 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:56.479472 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:56.479502 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 04:08:56.479576 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 04:08:56.479641 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":150,"threads_started":1}
I20260924 04:08:56.480360 7689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c75237d6b9e4f22a57b5afee30951f7" candidate_uuid: "dc6019f5f5474832b90b9150dcc4b0b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5a609a5eb2a40cfaf3f6616900bd505" is_pre_election: true
I20260924 04:08:56.480516 7689 raft_consensus.cc:2454] T 7c75237d6b9e4f22a57b5afee30951f7 P a5a609a5eb2a40cfaf3f6616900bd505 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc6019f5f5474832b90b9150dcc4b0b3 in term 0.
I20260924 04:08:56.480450 7552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c75237d6b9e4f22a57b5afee30951f7" candidate_uuid: "dc6019f5f5474832b90b9150dcc4b0b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67def1cdfdb048899741175fab806933" is_pre_election: true
I20260924 04:08:56.480727 7552 raft_consensus.cc:2454] T 7c75237d6b9e4f22a57b5afee30951f7 P 67def1cdfdb048899741175fab806933 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc6019f5f5474832b90b9150dcc4b0b3 in term 0.
I20260924 04:08:56.480952 7349 leader_election.cc:302] T 7c75237d6b9e4f22a57b5afee30951f7 P dc6019f5f5474832b90b9150dcc4b0b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5a609a5eb2a40cfaf3f6616900bd505, dc6019f5f5474832b90b9150dcc4b0b3; no voters:
I20260924 04:08:56.481249 7860 raft_consensus.cc:2781] T 7c75237d6b9e4f22a57b5afee30951f7 P dc6019f5f5474832b90b9150dcc4b0b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:08:56.481333 7860 raft_consensus.cc:482] T 7c75237d6b9e4f22a57b5afee30951f7 P dc6019f5f5474832b90b9150dcc4b0b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:08:56.481382 7860 raft_consensus.cc:3037] T 7c75237d6b9e4f22a57b5afee30951f7 P dc6019f5f5474832b90b9150dcc4b0b3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:08:56.482489 7860 raft_consensus.cc:504] T 7c75237d6b9e4f22a57b5afee30951f7 P dc6019f5f5474832b90b9150dcc4b0b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5a609a5eb2a40cfaf3f6616900bd505" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33301 } } peers { permanent_uuid: "67def1cdfdb048899741175fab806933" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37085 } } peers { permanent_uuid: "dc6019f5f5474832b90b9150dcc4b0b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44685 } }
I20260924 04:08:56.482764 7860 leader_election.cc:288] T 7c75237d6b9e4f22a57b5afee30951f7 P dc6019f5f5474832b90b9150dcc4b0b3 [CANDIDATE]: Term 1 election: Requested vote from peers a5a609a5eb2a40cfaf3f6616900bd505 (127.0.98.3:33301), 67def1cdfdb048899741175fab806933 (127.0.98.2:37085)
I20260924 04:08:56.483217 7689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c75237d6b9e4f22a57b5afee30951f7" candidate_uuid: "dc6019f5f5474832b90b9150dcc4b0b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5a609a5eb2a40cfaf3f6616900bd505"
I20260924 04:08:56.483328 7552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c75237d6b9e4f22a57b5afee30951f7" candidate_uuid: "dc6019f5f5474832b90b9150dcc4b0b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67def1cdfdb048899741175fab806933"
I20260924 04:08:56.483340 7689 raft_consensus.cc:3037] T 7c75237d6b9e4f22a57b5afee30951f7 P a5a609a5eb2a40cfaf3f6616900bd505 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:08:56.483456 7552 raft_consensus.cc:3037] T 7c75237d6b9e4f22a57b5afee30951f7 P 67def1cdfdb048899741175fab806933 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:08:56.484285 7689 raft_consensus.cc:2454] T 7c75237d6b9e4f22a57b5afee30951f7 P a5a609a5eb2a40cfaf3f6616900bd505 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc6019f5f5474832b90b9150dcc4b0b3 in term 1.
I20260924 04:08:56.484443 7552 raft_consensus.cc:2454] T 7c75237d6b9e4f22a57b5afee30951f7 P 67def1cdfdb048899741175fab806933 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc6019f5f5474832b90b9150dcc4b0b3 in term 1.
I20260924 04:08:56.484586 7349 leader_election.cc:302] T 7c75237d6b9e4f22a57b5afee30951f7 P dc6019f5f5474832b90b9150dcc4b0b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5a609a5eb2a40cfaf3f6616900bd505, dc6019f5f5474832b90b9150dcc4b0b3; no voters:
I20260924 04:08:56.484861 7860 raft_consensus.cc:2781] T 7c75237d6b9e4f22a57b5afee30951f7 P dc6019f5f5474832b90b9150dcc4b0b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:08:56.485095 7860 raft_consensus.cc:686] T 7c75237d6b9e4f22a57b5afee30951f7 P dc6019f5f5474832b90b9150dcc4b0b3 [term 1 LEADER]: Becoming Leader. State: Replica: dc6019f5f5474832b90b9150dcc4b0b3, State: Running, Role: LEADER
I20260924 04:08:56.485342 7860 consensus_queue.cc:237] T 7c75237d6b9e4f22a57b5afee30951f7 P dc6019f5f5474832b90b9150dcc4b0b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5a609a5eb2a40cfaf3f6616900bd505" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33301 } } peers { permanent_uuid: "67def1cdfdb048899741175fab806933" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37085 } } peers { permanent_uuid: "dc6019f5f5474832b90b9150dcc4b0b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44685 } }
I20260924 04:08:56.485841 7861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:56.485603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38949 (local address 127.0.98.1:59115)
0924 04:08:56.485711 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:56.485761 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260924 04:08:56.487680 7098 catalog_manager.cc:5495] T 7c75237d6b9e4f22a57b5afee30951f7 P dc6019f5f5474832b90b9150dcc4b0b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to dc6019f5f5474832b90b9150dcc4b0b3 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "dc6019f5f5474832b90b9150dcc4b0b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5a609a5eb2a40cfaf3f6616900bd505" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "67def1cdfdb048899741175fab806933" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc6019f5f5474832b90b9150dcc4b0b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44685 } health_report { overall_health: HEALTHY } } }
I20260924 04:08:56.547132 7851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:56.543580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54096 (local address 127.0.98.1:44685)
0924 04:08:56.543724 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:56.543728 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:56.543741 (+ 13us) server_negotiation.cc:407] Connection header received
0924 04:08:56.543814 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:56.543816 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:56.543869 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:56.543960 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:08:56.544182 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:56.544701 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:56.545508 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:56.545721 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:56.545799 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:56.546022 (+ 223us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:08:56.546486 (+ 464us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:08:56.546590 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:56.546814 (+ 224us) server_negotiation.cc:299] Negotiation successful
0924 04:08:56.546900 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 04:08:56.550134 7689 raft_consensus.cc:1261] T 7c75237d6b9e4f22a57b5afee30951f7 P a5a609a5eb2a40cfaf3f6616900bd505 [term 1 FOLLOWER]: Refusing update from remote peer dc6019f5f5474832b90b9150dcc4b0b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:08:56.550375 7552 raft_consensus.cc:1261] T 7c75237d6b9e4f22a57b5afee30951f7 P 67def1cdfdb048899741175fab806933 [term 1 FOLLOWER]: Refusing update from remote peer dc6019f5f5474832b90b9150dcc4b0b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:08:56.550647 7860 consensus_queue.cc:1035] T 7c75237d6b9e4f22a57b5afee30951f7 P dc6019f5f5474832b90b9150dcc4b0b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5a609a5eb2a40cfaf3f6616900bd505" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:08:56.550962 7862 consensus_queue.cc:1035] T 7c75237d6b9e4f22a57b5afee30951f7 P dc6019f5f5474832b90b9150dcc4b0b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "67def1cdfdb048899741175fab806933" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:08:56.552772 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:56.552443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38949 (local address 127.0.98.2:51147)
0924 04:08:56.552596 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:56.552663 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260924 04:08:56.552904 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:56.552542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38949 (local address 127.0.98.3:47781)
0924 04:08:56.552772 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:56.552827 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167}
I20260924 04:08:56.557075 7861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:56.556590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38949 (local address 127.0.98.1:41535)
0924 04:08:56.556924 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:56.556981 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":270}
I20260924 04:08:56.557556 7868 mvcc.cc:205] Tried to move back new op lower bound from 7332753148104089600 to 7332753147846897664. Current Snapshot: MvccSnapshot[applied={T|T < 7332753148104089600}]
I20260924 04:08:56.557888 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:56.557596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38949 (local address 127.0.98.3:38805)
0924 04:08:56.557766 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:56.557817 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260924 04:08:56.558300 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:56.557408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38949 (local address 127.0.98.2:45609)
0924 04:08:56.558176 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:56.558228 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":700}
I20260924 04:08:56.560760 7861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:56.560483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38949 (local address 127.0.98.1:53505)
0924 04:08:56.560620 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:56.560665 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260924 04:08:56.565414 7098 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42264:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:08:56.565582 7098 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42264:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:08:56.569615 7098 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3c01afc87f8b4560b973c517eea256a2: Sending DeleteTablet for 3 replicas of tablet 7c75237d6b9e4f22a57b5afee30951f7
I20260924 04:08:56.570220 7669 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c75237d6b9e4f22a57b5afee30951f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:08:56 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:45954
I20260924 04:08:56.570314 7532 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c75237d6b9e4f22a57b5afee30951f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:08:56 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:43010
I20260924 04:08:56.570930 7395 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c75237d6b9e4f22a57b5afee30951f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:08:56 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:54080
I20260924 04:08:56.570986 7877 tablet_replica.cc:330] T 7c75237d6b9e4f22a57b5afee30951f7 P 67def1cdfdb048899741175fab806933: stopping tablet replica
I20260924 04:08:56.571133 7877 raft_consensus.cc:2229] T 7c75237d6b9e4f22a57b5afee30951f7 P 67def1cdfdb048899741175fab806933 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:08:56.571301 7877 raft_consensus.cc:2258] T 7c75237d6b9e4f22a57b5afee30951f7 P 67def1cdfdb048899741175fab806933 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:08:56.571419 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 7243
I20260924 04:08:56.573818 7877 ts_tablet_manager.cc:1905] T 7c75237d6b9e4f22a57b5afee30951f7 P 67def1cdfdb048899741175fab806933: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:08:56.574457 7876 tablet_replica.cc:330] T 7c75237d6b9e4f22a57b5afee30951f7 P a5a609a5eb2a40cfaf3f6616900bd505: stopping tablet replica
I20260924 04:08:56.574638 7876 raft_consensus.cc:2229] T 7c75237d6b9e4f22a57b5afee30951f7 P a5a609a5eb2a40cfaf3f6616900bd505 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:08:56.574801 7876 raft_consensus.cc:2258] T 7c75237d6b9e4f22a57b5afee30951f7 P a5a609a5eb2a40cfaf3f6616900bd505 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:08:56.575542 7876 ts_tablet_manager.cc:1905] T 7c75237d6b9e4f22a57b5afee30951f7 P a5a609a5eb2a40cfaf3f6616900bd505: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:08:56.576478 7877 ts_tablet_manager.cc:1918] T 7c75237d6b9e4f22a57b5afee30951f7 P 67def1cdfdb048899741175fab806933: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:08:56.576572 7877 log.cc:1196] T 7c75237d6b9e4f22a57b5afee30951f7 P 67def1cdfdb048899741175fab806933: Deleting WAL directory at /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/ts-1/wal/wals/7c75237d6b9e4f22a57b5afee30951f7
I20260924 04:08:56.576882 7877 ts_tablet_manager.cc:1939] T 7c75237d6b9e4f22a57b5afee30951f7 P 67def1cdfdb048899741175fab806933: Deleting consensus metadata
I20260924 04:08:56.577587 7084 catalog_manager.cc:4841] TS 67def1cdfdb048899741175fab806933 (127.0.98.2:37085): tablet 7c75237d6b9e4f22a57b5afee30951f7 (table test-table [id=08a650465565473dac65e340ecdccd2e]) successfully deleted
I20260924 04:08:56.578749 7876 ts_tablet_manager.cc:1918] T 7c75237d6b9e4f22a57b5afee30951f7 P a5a609a5eb2a40cfaf3f6616900bd505: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:08:56.578828 7876 log.cc:1196] T 7c75237d6b9e4f22a57b5afee30951f7 P a5a609a5eb2a40cfaf3f6616900bd505: Deleting WAL directory at /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790222898807720-392-0/minicluster-data/ts-2/wal/wals/7c75237d6b9e4f22a57b5afee30951f7
I20260924 04:08:56.579092 7876 ts_tablet_manager.cc:1939] T 7c75237d6b9e4f22a57b5afee30951f7 P a5a609a5eb2a40cfaf3f6616900bd505: Deleting consensus metadata
I20260924 04:08:56.579303 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 7469
I20260924 04:08:56.579504 7849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:56.579232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44685 (local address 127.0.0.1:54102)
0924 04:08:56.579357 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:56.579413 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260924 04:08:56.580010 7084 catalog_manager.cc:4841] TS a5a609a5eb2a40cfaf3f6616900bd505 (127.0.98.3:33301): tablet 7c75237d6b9e4f22a57b5afee30951f7 (table test-table [id=08a650465565473dac65e340ecdccd2e]) successfully deleted
W20260924 04:08:56.580052 7086 catalog_manager.cc:4584] TS dc6019f5f5474832b90b9150dcc4b0b3 (127.0.98.1:44685): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7c75237d6b9e4f22a57b5afee30951f7: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44685: connect: Connection refused (error 111)
I20260924 04:08:56.586627 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 7605
I20260924 04:08:56.593700 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 6963
I20260924 04:08:56.599598 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 7071
2026-09-24T04:08:56Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (8057 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:08:56 dist-test-slave-00ld krb5kdc[7881](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:08:56 dist-test-slave-00ld krb5kdc[7881](info): set up 2 sockets
Sep 24 04:08:56 dist-test-slave-00ld krb5kdc[7881](info): commencing operation
krb5kdc: starting...
W20260924 04:08:57.645449 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/krb5kdc/test-user.keytab.
Sep 24 04:08:57 dist-test-slave-00ld krb5kdc[7881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222937, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:08:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:08:57Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260924 04:08:57.769850 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32951
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32951
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:08:57.884336 7894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:08:57.884636 7894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:08:57.884733 7894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:08:57.888808 7894 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:08:57.888877 7894 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 04:08:57.888896 7894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:08:57.888914 7894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:08:57.888931 7894 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:08:57.888947 7894 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:08:57.894333 7894 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32951
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32951
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7894
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:08:57.895375 7894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:08:57.896353 7894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:08:57.904615 7900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:08:57.904806 7894 server_base.cc:1034] running on GCE node
W20260924 04:08:57.904621 7902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:08:57.904637 7899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:08:57.905584 7894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:08:57.906776 7894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:08:57.908041 7894 hybrid_clock.cc:648] HybridClock initialized: now 1790222937908025 us; error 34 us; skew 500 ppm
Sep 24 04:08:57 dist-test-slave-00ld krb5kdc[7881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222937, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:08:57.910787 7894 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260924 04:08:57.911657 7894 webserver.cc:460] Webserver started at http://127.0.98.62:35421/ using document root <none> and password file <none>
I20260924 04:08:57.912246 7894 fs_manager.cc:359] Metadata directory not provided
I20260924 04:08:57.912304 7894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:08:57.912483 7894 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:08:57.914556 7894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/minicluster-data/master-0/data/instance:
uuid: "37584f0cfe7049d291a195adfd8d7aaf"
format_stamp: "Formatted at 2026-09-24 04:08:57 on dist-test-slave-00ld"
server_key: "f000b0949d5a1fc0e887efb531b43fee"
server_key_iv: "a4880bc108c35901bf519b7ffa76e87b"
server_key_version: "encryptionkey@0"
I20260924 04:08:57.915055 7894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/minicluster-data/master-0/wal/instance:
uuid: "37584f0cfe7049d291a195adfd8d7aaf"
format_stamp: "Formatted at 2026-09-24 04:08:57 on dist-test-slave-00ld"
server_key: "f000b0949d5a1fc0e887efb531b43fee"
server_key_iv: "a4880bc108c35901bf519b7ffa76e87b"
server_key_version: "encryptionkey@0"
I20260924 04:08:57.918855 7894 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 04:08:57.921167 7909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:57.922356 7894 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:08:57.922475 7894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/minicluster-data/master-0/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/minicluster-data/master-0/wal
uuid: "37584f0cfe7049d291a195adfd8d7aaf"
format_stamp: "Formatted at 2026-09-24 04:08:57 on dist-test-slave-00ld"
server_key: "f000b0949d5a1fc0e887efb531b43fee"
server_key_iv: "a4880bc108c35901bf519b7ffa76e87b"
server_key_version: "encryptionkey@0"
I20260924 04:08:57.922577 7894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:08:57.932660 7894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:08:57.935367 7894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:08:57.935556 7894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:08:57.943140 7894 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:32951
I20260924 04:08:57.943169 7971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:32951 every 8 connection(s)
I20260924 04:08:57.944052 7894 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:08:57.945981 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 7894
I20260924 04:08:57.946105 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/minicluster-data/master-0/wal/instance
I20260924 04:08:57.946421 392 external_mini_cluster.cc:1428] Setting key da2a9abeb77035eac2adc59f1b9e15c4
I20260924 04:08:57.947049 7972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:08:57.952800 7972 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37584f0cfe7049d291a195adfd8d7aaf: Bootstrap starting.
Sep 24 04:08:57 dist-test-slave-00ld krb5kdc[7881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222937, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 04:08:57.955281 7972 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 37584f0cfe7049d291a195adfd8d7aaf: Neither blocks nor log segments found. Creating new log.
I20260924 04:08:57.956063 7972 log.cc:824] T 00000000000000000000000000000000 P 37584f0cfe7049d291a195adfd8d7aaf: Log is configured to *not* fsync() on all Append() calls
I20260924 04:08:57.958115 7972 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37584f0cfe7049d291a195adfd8d7aaf: No bootstrap required, opened a new log
I20260924 04:08:57.960888 7975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:57.947823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35008 (local address 127.0.98.62:32951)
0924 04:08:57.948236 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:57.948250 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:57.948285 (+ 35us) server_negotiation.cc:407] Connection header received
0924 04:08:57.949002 (+ 717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:57.949019 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:57.949356 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:57.949683 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:57.950459 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:57.951421 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:57.952240 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:57.952766 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:57.952913 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:57.955539 (+ 2626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:57.955544 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:57.955556 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:57.955581 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:57.957787 (+ 2206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:57.958300 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:57.958308 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:57.958315 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:57.958470 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:57.958893 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:57.958899 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:57.958903 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:57.959346 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:57.959477 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:57.959795 (+ 318us) server_negotiation.cc:299] Negotiation successful
0924 04:08:57.960066 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":147,"threads_started":1}
I20260924 04:08:57.961647 7972 raft_consensus.cc:348] T 00000000000000000000000000000000 P 37584f0cfe7049d291a195adfd8d7aaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37584f0cfe7049d291a195adfd8d7aaf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32951 } }
I20260924 04:08:57.961930 7972 raft_consensus.cc:374] T 00000000000000000000000000000000 P 37584f0cfe7049d291a195adfd8d7aaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:08:57.961992 7972 raft_consensus.cc:729] T 00000000000000000000000000000000 P 37584f0cfe7049d291a195adfd8d7aaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37584f0cfe7049d291a195adfd8d7aaf, State: Initialized, Role: FOLLOWER
I20260924 04:08:57.962458 7972 consensus_queue.cc:260] T 00000000000000000000000000000000 P 37584f0cfe7049d291a195adfd8d7aaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37584f0cfe7049d291a195adfd8d7aaf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32951 } }
I20260924 04:08:57.962610 7972 raft_consensus.cc:388] T 00000000000000000000000000000000 P 37584f0cfe7049d291a195adfd8d7aaf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:08:57.962677 7972 raft_consensus.cc:482] T 00000000000000000000000000000000 P 37584f0cfe7049d291a195adfd8d7aaf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:08:57.962769 7972 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 37584f0cfe7049d291a195adfd8d7aaf [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:08:57.963714 7972 raft_consensus.cc:504] T 00000000000000000000000000000000 P 37584f0cfe7049d291a195adfd8d7aaf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37584f0cfe7049d291a195adfd8d7aaf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32951 } }
I20260924 04:08:57.964005 7972 leader_election.cc:302] T 00000000000000000000000000000000 P 37584f0cfe7049d291a195adfd8d7aaf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 37584f0cfe7049d291a195adfd8d7aaf; no voters:
I20260924 04:08:57.964251 7972 leader_election.cc:288] T 00000000000000000000000000000000 P 37584f0cfe7049d291a195adfd8d7aaf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:08:57.964363 7977 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 37584f0cfe7049d291a195adfd8d7aaf [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:08:57.964609 7977 raft_consensus.cc:686] T 00000000000000000000000000000000 P 37584f0cfe7049d291a195adfd8d7aaf [term 1 LEADER]: Becoming Leader. State: Replica: 37584f0cfe7049d291a195adfd8d7aaf, State: Running, Role: LEADER
I20260924 04:08:57.965006 7977 consensus_queue.cc:237] T 00000000000000000000000000000000 P 37584f0cfe7049d291a195adfd8d7aaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37584f0cfe7049d291a195adfd8d7aaf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32951 } }
I20260924 04:08:57.965590 7972 sys_catalog.cc:564] T 00000000000000000000000000000000 P 37584f0cfe7049d291a195adfd8d7aaf [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:08:57.966739 7978 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37584f0cfe7049d291a195adfd8d7aaf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "37584f0cfe7049d291a195adfd8d7aaf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37584f0cfe7049d291a195adfd8d7aaf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32951 } } }
I20260924 04:08:57.966881 7978 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37584f0cfe7049d291a195adfd8d7aaf [sys.catalog]: This master's current role is: LEADER
I20260924 04:08:57.967284 7986 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:08:57.967674 7979 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37584f0cfe7049d291a195adfd8d7aaf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37584f0cfe7049d291a195adfd8d7aaf. Latest consensus state: current_term: 1 leader_uuid: "37584f0cfe7049d291a195adfd8d7aaf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37584f0cfe7049d291a195adfd8d7aaf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32951 } } }
I20260924 04:08:57.967774 7979 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37584f0cfe7049d291a195adfd8d7aaf [sys.catalog]: This master's current role is: LEADER
I20260924 04:08:57.970135 7986 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:08:57.975714 7986 catalog_manager.cc:1347] Generated new cluster ID: effa4f0ff13f4ee7bef8ae55ec6cf650
I20260924 04:08:57.975782 7986 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:08:58.005527 7986 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:08:58.006472 7986 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:08:58.013244 7986 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 37584f0cfe7049d291a195adfd8d7aaf: Generated new TSK 0
I20260924 04:08:58.013940 7986 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260924 04:08:58.051957 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32951
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:08:58.163419 7998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:08:58.163726 7998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:08:58.163805 7998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:08:58.168004 7998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:08:58.168085 7998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:08:58.168155 7998 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 04:08:58.168228 7998 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:08:58.173449 7998 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:32951
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7998
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:08:58.174557 7998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:08:58.175607 7998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:08:58.185122 8006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:08:58.185081 8003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:08:58.185099 8004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:08:58.185226 7998 server_base.cc:1034] running on GCE node
I20260924 04:08:58.185839 7998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:08:58.186519 7998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:08:58.187702 7998 hybrid_clock.cc:648] HybridClock initialized: now 1790222938187689 us; error 41 us; skew 500 ppm
Sep 24 04:08:58 dist-test-slave-00ld krb5kdc[7881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222938, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:08:58.197172 7998 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260924 04:08:58.198097 7998 webserver.cc:460] Webserver started at http://127.0.98.1:46207/ using document root <none> and password file <none>
I20260924 04:08:58.198685 7998 fs_manager.cc:359] Metadata directory not provided
I20260924 04:08:58.198770 7998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:08:58.198994 7998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:08:58.201089 7998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/minicluster-data/ts-0/data/instance:
uuid: "cc944248f10b48bc98c503b0f733a0e9"
format_stamp: "Formatted at 2026-09-24 04:08:58 on dist-test-slave-00ld"
server_key: "178d6c9d22d303d90221045290ab1b4f"
server_key_iv: "967b80da096077594e9033c70e8bb2ca"
server_key_version: "encryptionkey@0"
I20260924 04:08:58.201624 7998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/minicluster-data/ts-0/wal/instance:
uuid: "cc944248f10b48bc98c503b0f733a0e9"
format_stamp: "Formatted at 2026-09-24 04:08:58 on dist-test-slave-00ld"
server_key: "178d6c9d22d303d90221045290ab1b4f"
server_key_iv: "967b80da096077594e9033c70e8bb2ca"
server_key_version: "encryptionkey@0"
I20260924 04:08:58.205247 7998 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:08:58.207576 8013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:58.208809 7998 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:08:58.208925 7998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/minicluster-data/ts-0/wal
uuid: "cc944248f10b48bc98c503b0f733a0e9"
format_stamp: "Formatted at 2026-09-24 04:08:58 on dist-test-slave-00ld"
server_key: "178d6c9d22d303d90221045290ab1b4f"
server_key_iv: "967b80da096077594e9033c70e8bb2ca"
server_key_version: "encryptionkey@0"
I20260924 04:08:58.209023 7998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:08:58.229038 7998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:08:58.232191 7998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:08:58.232394 7998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:08:58.233909 7998 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:08:58.233989 7998 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:58.234056 7998 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:08:58.234107 7998 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 04:08:58 dist-test-slave-00ld krb5kdc[7881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222938, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 04:08:58.244825 7998 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45551
I20260924 04:08:58.244887 8128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45551 every 8 connection(s)
I20260924 04:08:58.245818 7998 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:08:58.247561 7975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:58.234722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38151 (local address 127.0.98.62:32951)
0924 04:08:58.234886 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:58.234893 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:58.235698 (+ 805us) server_negotiation.cc:407] Connection header received
0924 04:08:58.236799 (+ 1101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:58.236803 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:58.236861 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:58.236948 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:58.238154 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:58.238836 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:58.240010 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:58.240261 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:58.240365 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:58.243949 (+ 3584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:58.243953 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:58.243960 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:58.243985 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:58.245571 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:58.246121 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:58.246124 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:58.246126 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:58.246206 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:58.246584 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:58.246590 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:58.246592 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:58.246787 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:58.246870 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:58.247335 (+ 465us) server_negotiation.cc:299] Negotiation successful
0924 04:08:58.247439 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 04:08:58.248385 8022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:58.235102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32951 (local address 127.0.98.1:38151)
0924 04:08:58.235536 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:58.235584 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:58.236556 (+ 972us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:58.237140 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:58.237148 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:58.237476 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:08:58.238015 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:58.238031 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:58.238953 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:58.238957 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:58.239895 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:58.239904 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:58.240403 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:58.240407 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:58.240621 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:58.241288 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:08:58.241312 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:08:58.243785 (+ 2473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:08:58.245681 (+ 1896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:58.245686 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:58.245700 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:58.246024 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:58.246301 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:58.246305 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:58.246307 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:58.246479 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:58.246890 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:08:58.246898 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:08:58.247132 (+ 234us) client_negotiation.cc:769] Sending connection context
0924 04:08:58.247283 (+ 151us) client_negotiation.cc:240] Negotiation successful
0924 04:08:58.247516 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":113,"threads_started":1}
I20260924 04:08:58.248765 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 7998
I20260924 04:08:58.248893 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:08:58.249159 392 external_mini_cluster.cc:1428] Setting key 3da746b708f929f3280b2e78ba813165
I20260924 04:08:58.249830 8129 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32951
I20260924 04:08:58.250010 8129 heartbeater.cc:464] Registering TS with master...
I20260924 04:08:58.250590 8129 heartbeater.cc:511] Master 127.0.98.62:32951 requested a full tablet report, sending...
I20260924 04:08:58.252276 7925 ts_manager.cc:194] Registered new tserver with Master: cc944248f10b48bc98c503b0f733a0e9 (127.0.98.1:45551)
I20260924 04:08:58.253923 7925 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38151
I20260924 04:08:58.260023 7975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:58.252174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59997 (local address 127.0.98.62:32951)
0924 04:08:58.252282 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:58.252287 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:58.252594 (+ 307us) server_negotiation.cc:407] Connection header received
0924 04:08:58.252785 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:58.252788 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:58.252835 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:58.252938 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:58.253347 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:58.253861 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:58.254949 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:58.255370 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:58.255456 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:58.257023 (+ 1567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:58.257027 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:58.257030 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:58.257055 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:58.258460 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:58.258915 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:58.258917 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:58.258918 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:58.258987 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:58.259278 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:58.259280 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:58.259281 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:58.259438 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:58.259509 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:58.259812 (+ 303us) server_negotiation.cc:299] Negotiation successful
0924 04:08:58.259907 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 04:08:58.260169 8135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:58.252249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32951 (local address 127.0.98.1:59997)
0924 04:08:58.252514 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:58.252526 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:58.252636 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:58.252946 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:58.252949 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:58.253151 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:08:58.253242 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:58.253247 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:58.253988 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:58.253991 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:58.254811 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:58.254825 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:58.255623 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:58.255629 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:58.255757 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:58.256333 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:08:58.256350 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:08:58.256885 (+ 535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:08:58.258568 (+ 1683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:58.258573 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:58.258575 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:58.258822 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:58.259074 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:58.259080 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:58.259084 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:58.259191 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:58.259533 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:08:58.259537 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:08:58.259660 (+ 123us) client_negotiation.cc:769] Sending connection context
0924 04:08:58.259775 (+ 115us) client_negotiation.cc:240] Negotiation successful
0924 04:08:58.259918 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":112,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260924 04:08:58.279433 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32951
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:08:58.393702 8138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:08:58.393991 8138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:08:58.394073 8138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:08:58.398213 8138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:08:58.398319 8138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:08:58.398380 8138 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 04:08:58.398422 8138 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:08:58.403712 8138 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:32951
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8138
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:08:58.404850 8138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:08:58.405871 8138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:08:58.415728 8143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:08:58.415729 8138 server_base.cc:1034] running on GCE node
W20260924 04:08:58.415732 8144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:08:58.415745 8146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:08:58.416429 8138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:08:58.417143 8138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:08:58.418329 8138 hybrid_clock.cc:648] HybridClock initialized: now 1790222938418311 us; error 31 us; skew 500 ppm
Sep 24 04:08:58 dist-test-slave-00ld krb5kdc[7881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222938, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:08:58.421077 8138 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260924 04:08:58.421916 8138 webserver.cc:460] Webserver started at http://127.0.98.2:33245/ using document root <none> and password file <none>
I20260924 04:08:58.422513 8138 fs_manager.cc:359] Metadata directory not provided
I20260924 04:08:58.422605 8138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:08:58.422834 8138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:08:58.424912 8138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/minicluster-data/ts-1/data/instance:
uuid: "61447010247c4936bc9a9fcb2af3d202"
format_stamp: "Formatted at 2026-09-24 04:08:58 on dist-test-slave-00ld"
server_key: "e1973921fbdbed3aa2703ab1e1435756"
server_key_iv: "06a5e0a2f81bf6387b9ae4c93d448ab0"
server_key_version: "encryptionkey@0"
I20260924 04:08:58.425477 8138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/minicluster-data/ts-1/wal/instance:
uuid: "61447010247c4936bc9a9fcb2af3d202"
format_stamp: "Formatted at 2026-09-24 04:08:58 on dist-test-slave-00ld"
server_key: "e1973921fbdbed3aa2703ab1e1435756"
server_key_iv: "06a5e0a2f81bf6387b9ae4c93d448ab0"
server_key_version: "encryptionkey@0"
I20260924 04:08:58.429198 8138 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260924 04:08:58.431699 8153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:58.432919 8138 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:08:58.433041 8138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/minicluster-data/ts-1/wal
uuid: "61447010247c4936bc9a9fcb2af3d202"
format_stamp: "Formatted at 2026-09-24 04:08:58 on dist-test-slave-00ld"
server_key: "e1973921fbdbed3aa2703ab1e1435756"
server_key_iv: "06a5e0a2f81bf6387b9ae4c93d448ab0"
server_key_version: "encryptionkey@0"
I20260924 04:08:58.433161 8138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:08:58.448140 8138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:08:58.451784 8138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:08:58.452013 8138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:08:58.453656 8138 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:08:58.453729 8138 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:58.453806 8138 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:08:58.453847 8138 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 04:08:58 dist-test-slave-00ld krb5kdc[7881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222938, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 04:08:58.465078 8138 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45391
I20260924 04:08:58.465116 8268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45391 every 8 connection(s)
I20260924 04:08:58.466532 8138 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:08:58.467667 7975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:58.454483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60605 (local address 127.0.98.62:32951)
0924 04:08:58.454611 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:58.454615 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:58.455713 (+ 1098us) server_negotiation.cc:407] Connection header received
0924 04:08:58.456845 (+ 1132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:58.456848 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:58.456920 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:58.457071 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:58.458452 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:58.459043 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:58.460449 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:58.460773 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:58.460893 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:58.463476 (+ 2583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:58.463483 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:58.463489 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:58.463527 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:58.465303 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:58.465927 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:58.465930 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:58.465932 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:58.466036 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:58.466375 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:58.466378 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:58.466380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:58.466637 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:58.466757 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:58.467386 (+ 629us) server_negotiation.cc:299] Negotiation successful
0924 04:08:58.467513 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 04:08:58.468559 8162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:58.455048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32951 (local address 127.0.98.2:60605)
0924 04:08:58.455564 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:58.455607 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:58.456596 (+ 989us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:58.457361 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:58.457368 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:58.457704 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:08:58.458288 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:58.458319 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:58.459162 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:58.459169 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:58.460287 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:58.460300 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:58.460955 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:58.460960 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:58.461123 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:58.461747 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:08:58.461764 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:08:58.463298 (+ 1534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:08:58.465411 (+ 2113us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:58.465416 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:58.465433 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:58.465804 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:58.466130 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:58.466133 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:58.466134 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:58.466268 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:58.466789 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:08:58.466796 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:08:58.467133 (+ 337us) client_negotiation.cc:769] Sending connection context
0924 04:08:58.467339 (+ 206us) client_negotiation.cc:240] Negotiation successful
0924 04:08:58.467673 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":124,"threads_started":1}
I20260924 04:08:58.469899 8269 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32951
I20260924 04:08:58.470052 8269 heartbeater.cc:464] Registering TS with master...
I20260924 04:08:58.470600 8269 heartbeater.cc:511] Master 127.0.98.62:32951 requested a full tablet report, sending...
I20260924 04:08:58.471626 7925 ts_manager.cc:194] Registered new tserver with Master: 61447010247c4936bc9a9fcb2af3d202 (127.0.98.2:45391)
I20260924 04:08:58.472368 7925 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60605
I20260924 04:08:58.475816 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 8138
I20260924 04:08:58.475948 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:08:58.476311 392 external_mini_cluster.cc:1428] Setting key cbbd130bd1f1c710885a109bcb697d7c
I20260924 04:08:58.479244 8275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:58.471966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32951 (local address 127.0.98.2:43347)
0924 04:08:58.472234 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:58.472249 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:58.472345 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:58.472604 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:58.472609 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:58.472892 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:08:58.473026 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:58.473034 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:58.474012 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:58.474014 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:58.474576 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:58.474584 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:58.474975 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:58.474978 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:58.475045 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:58.475498 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:08:58.475509 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:08:58.475811 (+ 302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:08:58.477751 (+ 1940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:58.477754 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:58.477755 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:58.477942 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:58.478232 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:58.478234 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:58.478235 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:58.478307 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:58.478815 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:08:58.478817 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:08:58.478893 (+ 76us) client_negotiation.cc:769] Sending connection context
0924 04:08:58.479002 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 04:08:58.479111 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":108,"threads_started":1}
I20260924 04:08:58.479450 7975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:58.472024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43347 (local address 127.0.98.62:32951)
0924 04:08:58.472150 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:58.472155 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:58.472316 (+ 161us) server_negotiation.cc:407] Connection header received
0924 04:08:58.472428 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:58.472430 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:58.472493 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:58.472566 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:58.473131 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:58.473872 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:58.474674 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:58.474877 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:58.474954 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:58.475983 (+ 1029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:58.475987 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:58.475990 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:58.476017 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:58.477652 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:58.478040 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:58.478044 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:58.478046 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:58.478138 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:58.478391 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:58.478397 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:58.478402 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:58.478715 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:58.478825 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:58.479080 (+ 255us) server_negotiation.cc:299] Negotiation successful
0924 04:08:58.479209 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260924 04:08:58.503844 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32951
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:08:58.614141 8278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:08:58.614418 8278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:08:58.614492 8278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:08:58.618501 8278 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:08:58.618590 8278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:08:58.618659 8278 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 04:08:58.618737 8278 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:08:58.624413 8278 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:32951
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8278
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:08:58.625442 8278 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:08:58.626443 8278 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:08:58.635844 8278 server_base.cc:1034] running on GCE node
W20260924 04:08:58.635732 8283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:08:58.635727 8286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:08:58.635785 8284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:08:58.636466 8278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:08:58.637203 8278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:08:58.638405 8278 hybrid_clock.cc:648] HybridClock initialized: now 1790222938638388 us; error 37 us; skew 500 ppm
Sep 24 04:08:58 dist-test-slave-00ld krb5kdc[7881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222938, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:08:58.641346 8278 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260924 04:08:58.642238 8278 webserver.cc:460] Webserver started at http://127.0.98.3:43733/ using document root <none> and password file <none>
I20260924 04:08:58.642794 8278 fs_manager.cc:359] Metadata directory not provided
I20260924 04:08:58.642854 8278 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:08:58.643025 8278 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:08:58.645208 8278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/minicluster-data/ts-2/data/instance:
uuid: "14aa376377e34cae9a387069ec429d4a"
format_stamp: "Formatted at 2026-09-24 04:08:58 on dist-test-slave-00ld"
server_key: "bc047ef7dfa8d33ed46c495f13addd15"
server_key_iv: "e631a71b8fb712d9496ead84f07901ec"
server_key_version: "encryptionkey@0"
I20260924 04:08:58.645750 8278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/minicluster-data/ts-2/wal/instance:
uuid: "14aa376377e34cae9a387069ec429d4a"
format_stamp: "Formatted at 2026-09-24 04:08:58 on dist-test-slave-00ld"
server_key: "bc047ef7dfa8d33ed46c495f13addd15"
server_key_iv: "e631a71b8fb712d9496ead84f07901ec"
server_key_version: "encryptionkey@0"
I20260924 04:08:58.649586 8278 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 04:08:58.652034 8293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:58.653376 8278 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:08:58.653506 8278 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/minicluster-data/ts-2/wal
uuid: "14aa376377e34cae9a387069ec429d4a"
format_stamp: "Formatted at 2026-09-24 04:08:58 on dist-test-slave-00ld"
server_key: "bc047ef7dfa8d33ed46c495f13addd15"
server_key_iv: "e631a71b8fb712d9496ead84f07901ec"
server_key_version: "encryptionkey@0"
I20260924 04:08:58.653625 8278 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:08:58.671491 8278 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:08:58.674700 8278 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:08:58.674961 8278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:08:58.676519 8278 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:08:58.676575 8278 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:08:58.676640 8278 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:08:58.676672 8278 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 04:08:58 dist-test-slave-00ld krb5kdc[7881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222938, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 04:08:58.687584 8278 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42625
I20260924 04:08:58.687688 8408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42625 every 8 connection(s)
I20260924 04:08:58.688580 8278 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:08:58.690004 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 8278
I20260924 04:08:58.690142 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:08:58.690490 392 external_mini_cluster.cc:1428] Setting key 962e54ddf582f914fe4663753987f73f
I20260924 04:08:58.690742 7975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:58.677471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49497 (local address 127.0.98.62:32951)
0924 04:08:58.677644 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:58.677649 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:58.678461 (+ 812us) server_negotiation.cc:407] Connection header received
0924 04:08:58.679532 (+ 1071us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:58.679536 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:58.679618 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:58.679687 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:58.681367 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:58.681937 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:58.683075 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:58.683344 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:58.683472 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:58.686842 (+ 3370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:58.686844 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:58.686846 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:58.686870 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:58.688459 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:58.689068 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:58.689073 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:58.689074 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:58.689162 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:58.689554 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:58.689558 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:58.689559 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:58.689766 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:58.689847 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:58.690473 (+ 626us) server_negotiation.cc:299] Negotiation successful
0924 04:08:58.690611 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 04:08:58.691761 8302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:58.677855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32951 (local address 127.0.98.3:49497)
0924 04:08:58.678289 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:58.678339 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:58.679255 (+ 916us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:58.680155 (+ 900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:58.680164 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:58.680617 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:08:58.681229 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:58.681242 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:58.682059 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:58.682063 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:58.682952 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:58.682961 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:58.683560 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:58.683568 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:58.683818 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:58.684558 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:08:58.684578 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:08:58.686670 (+ 2092us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:08:58.688572 (+ 1902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:58.688578 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:58.688593 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:58.688956 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:58.689274 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:58.689278 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:58.689280 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:58.689443 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:58.689869 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:08:58.689875 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:08:58.690213 (+ 338us) client_negotiation.cc:769] Sending connection context
0924 04:08:58.690418 (+ 205us) client_negotiation.cc:240] Negotiation successful
0924 04:08:58.690742 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":116,"threads_started":1}
I20260924 04:08:58.693042 8409 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32951
I20260924 04:08:58.693188 8409 heartbeater.cc:464] Registering TS with master...
I20260924 04:08:58.693699 8409 heartbeater.cc:511] Master 127.0.98.62:32951 requested a full tablet report, sending...
I20260924 04:08:58.694638 7926 ts_manager.cc:194] Registered new tserver with Master: 14aa376377e34cae9a387069ec429d4a (127.0.98.3:42625)
I20260924 04:08:58.695379 7926 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49497
I20260924 04:08:58.701287 7975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:58.695027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45825 (local address 127.0.98.62:32951)
0924 04:08:58.695158 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:58.695163 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:58.695258 (+ 95us) server_negotiation.cc:407] Connection header received
0924 04:08:58.695373 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:58.695375 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:58.695407 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:58.695492 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:58.695907 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:58.696403 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:58.697266 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:58.697497 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:58.697608 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:58.698556 (+ 948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:58.698560 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:58.698562 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:58.698584 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:58.699729 (+ 1145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:58.700126 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:58.700131 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:58.700133 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:58.700209 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:58.700465 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:58.700469 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:58.700471 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:58.700636 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:58.700788 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:58.701034 (+ 246us) server_negotiation.cc:299] Negotiation successful
0924 04:08:58.701137 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 04:08:58.701294 8415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:58.694917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32951 (local address 127.0.98.3:45825)
0924 04:08:58.695195 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:58.695209 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:58.695296 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:58.695508 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:58.695511 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:58.695707 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:08:58.695800 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:58.695806 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:58.696521 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:58.696526 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:58.697152 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:58.697160 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:58.697608 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:58.697611 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:58.697707 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:58.698124 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:08:58.698135 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:08:58.698428 (+ 293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:08:58.699834 (+ 1406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:58.699838 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:58.699840 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:58.700034 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:58.700301 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:58.700304 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:58.700306 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:58.700382 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:58.700778 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:08:58.700781 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:08:58.700873 (+ 92us) client_negotiation.cc:769] Sending connection context
0924 04:08:58.700972 (+ 99us) client_negotiation.cc:240] Negotiation successful
0924 04:08:58.701114 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":94,"threads_started":1}
I20260924 04:08:58.704068 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 04:08:58 dist-test-slave-00ld krb5kdc[7881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222938, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 04:08:58 dist-test-slave-00ld krb5kdc[7881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:08:58 dist-test-slave-00ld krb5kdc[7881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 04:08:58.726999 7975 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:08:58.720896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35014 (local address 127.0.98.62:32951)
0924 04:08:58.721052 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:58.721055 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:58.721067 (+ 12us) server_negotiation.cc:407] Connection header received
0924 04:08:58.721104 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:58.721107 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:58.721144 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:58.721233 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:58.721918 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:58.722447 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:58.723301 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:58.723513 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:58.723616 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:58.726831 (+ 3215us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35014: BlockingRecv error: recv got EOF from 127.0.0.1:35014 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 24 04:08:58 dist-test-slave-00ld krb5kdc[7881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222938, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 04:08:58.736292 7975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:58.728456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35020 (local address 127.0.98.62:32951)
0924 04:08:58.728599 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:58.728603 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:58.728748 (+ 145us) server_negotiation.cc:407] Connection header received
0924 04:08:58.728884 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:58.728887 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:58.728931 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:58.729043 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:58.729577 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:58.730063 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:58.730830 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:58.731045 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:58.731147 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:58.733391 (+ 2244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:58.733395 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:58.733396 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:58.733417 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:58.734714 (+ 1297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:58.735118 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:58.735122 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:58.735124 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:58.735199 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:58.735453 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:58.735458 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:58.735459 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:58.735626 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:58.735758 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:58.735981 (+ 223us) server_negotiation.cc:299] Negotiation successful
0924 04:08:58.736155 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 04:08:58.738955 7926 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35020:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:08:58.741925 7926 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:08:58.756017 8433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:58.750064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57816 (local address 127.0.98.3:42625)
0924 04:08:58.750616 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:58.750628 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:58.751006 (+ 378us) server_negotiation.cc:407] Connection header received
0924 04:08:58.751204 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:58.751209 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:58.751362 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:58.751481 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:08:58.751903 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:58.752814 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:58.754658 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:58.755375 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:58.755482 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:58.755645 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:58.755761 (+ 116us) server_negotiation.cc:299] Negotiation successful
0924 04:08:58.755857 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":94,"threads_started":1}
I20260924 04:08:58.756006 8434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:58.749892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42625 (local address 127.0.0.1:57816)
0924 04:08:58.750911 (+ 1019us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:58.750925 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:58.751075 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:58.751662 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:58.751665 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:58.751679 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:08:58.751780 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:58.751785 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:58.752954 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:58.752957 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:58.754552 (+ 1595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:58.754558 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:58.755507 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:58.755510 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:58.755584 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:58.755614 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 04:08:58.755729 (+ 115us) client_negotiation.cc:240] Negotiation successful
0924 04:08:58.755844 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":957,"mutex_wait_us":150,"thread_start_us":106,"threads_started":1}
I20260924 04:08:58.756734 8431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:58.750553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45391 (local address 127.0.0.1:35930)
0924 04:08:58.751257 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:58.751270 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:58.751423 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:58.752056 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:58.752058 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:58.752074 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:08:58.752165 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:58.752170 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:58.753345 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:58.753351 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:58.754428 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:58.754436 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:58.756309 (+ 1873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:58.756312 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:58.756412 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:58.756436 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 04:08:58.756512 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 04:08:58.756578 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"mutex_wait_us":24,"thread_start_us":89,"threads_started":1}
I20260924 04:08:58.757251 8432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:58.749790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45551 (local address 127.0.0.1:38806)
0924 04:08:58.750227 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:58.750270 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:58.750399 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:58.751495 (+ 1096us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:58.751503 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:58.751528 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:08:58.751654 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:58.751660 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:58.752904 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:58.752908 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:58.754546 (+ 1638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:58.754557 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:58.756822 (+ 2265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:58.756826 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:58.756926 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:58.756951 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 04:08:58.757023 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 04:08:58.757109 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"mutex_wait_us":121,"thread_start_us":142,"threads_started":1}
I20260924 04:08:58.757898 8435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:58.750616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38806 (local address 127.0.98.1:45551)
0924 04:08:58.750986 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:58.750993 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:58.751012 (+ 19us) server_negotiation.cc:407] Connection header received
0924 04:08:58.751100 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:58.751108 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:58.751317 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:58.751488 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:08:58.751776 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:58.752792 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:58.755708 (+ 2916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:58.756651 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:58.757523 (+ 872us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:58.757680 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:58.757727 (+ 47us) server_negotiation.cc:299] Negotiation successful
0924 04:08:58.757795 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":86,"threads_started":1}
I20260924 04:08:58.759272 8343 tablet_service.cc:1467] Processing CreateTablet for tablet ccb307eb26994fe7a8ef91556d0a9042 (DEFAULT_TABLE table=test-table [id=db3405d46f5f4cccb360fd94f3d62ff8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:08:58.760205 8343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccb307eb26994fe7a8ef91556d0a9042. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:08:58.760111 8063 tablet_service.cc:1467] Processing CreateTablet for tablet ccb307eb26994fe7a8ef91556d0a9042 (DEFAULT_TABLE table=test-table [id=db3405d46f5f4cccb360fd94f3d62ff8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:08:58.761050 8063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccb307eb26994fe7a8ef91556d0a9042. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:08:58.764969 8436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:58.750698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35930 (local address 127.0.98.2:45391)
0924 04:08:58.751139 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:58.751151 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:58.751336 (+ 185us) server_negotiation.cc:407] Connection header received
0924 04:08:58.751565 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:58.751570 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:58.751755 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:58.751891 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:08:58.752282 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:58.753244 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:58.754548 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:58.755375 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:58.755482 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:58.755645 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:58.757075 (+ 1430us) server_negotiation.cc:299] Negotiation successful
0924 04:08:58.764813 (+ 7738us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":107,"threads_started":1}
I20260924 04:08:58.767059 8437 tablet_bootstrap.cc:492] T ccb307eb26994fe7a8ef91556d0a9042 P cc944248f10b48bc98c503b0f733a0e9: Bootstrap starting.
I20260924 04:08:58.767812 8203 tablet_service.cc:1467] Processing CreateTablet for tablet ccb307eb26994fe7a8ef91556d0a9042 (DEFAULT_TABLE table=test-table [id=db3405d46f5f4cccb360fd94f3d62ff8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:08:58.768399 8438 tablet_bootstrap.cc:492] T ccb307eb26994fe7a8ef91556d0a9042 P 14aa376377e34cae9a387069ec429d4a: Bootstrap starting.
I20260924 04:08:58.768898 8203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccb307eb26994fe7a8ef91556d0a9042. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:08:58.769454 8437 tablet_bootstrap.cc:654] T ccb307eb26994fe7a8ef91556d0a9042 P cc944248f10b48bc98c503b0f733a0e9: Neither blocks nor log segments found. Creating new log.
I20260924 04:08:58.770434 8437 log.cc:824] T ccb307eb26994fe7a8ef91556d0a9042 P cc944248f10b48bc98c503b0f733a0e9: Log is configured to *not* fsync() on all Append() calls
I20260924 04:08:58.770614 8438 tablet_bootstrap.cc:654] T ccb307eb26994fe7a8ef91556d0a9042 P 14aa376377e34cae9a387069ec429d4a: Neither blocks nor log segments found. Creating new log.
I20260924 04:08:58.771373 8438 log.cc:824] T ccb307eb26994fe7a8ef91556d0a9042 P 14aa376377e34cae9a387069ec429d4a: Log is configured to *not* fsync() on all Append() calls
I20260924 04:08:58.772758 8437 tablet_bootstrap.cc:492] T ccb307eb26994fe7a8ef91556d0a9042 P cc944248f10b48bc98c503b0f733a0e9: No bootstrap required, opened a new log
I20260924 04:08:58.772979 8437 ts_tablet_manager.cc:1397] T ccb307eb26994fe7a8ef91556d0a9042 P cc944248f10b48bc98c503b0f733a0e9: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 04:08:58.773386 8438 tablet_bootstrap.cc:492] T ccb307eb26994fe7a8ef91556d0a9042 P 14aa376377e34cae9a387069ec429d4a: No bootstrap required, opened a new log
I20260924 04:08:58.773567 8438 ts_tablet_manager.cc:1397] T ccb307eb26994fe7a8ef91556d0a9042 P 14aa376377e34cae9a387069ec429d4a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 04:08:58.776415 8441 tablet_bootstrap.cc:492] T ccb307eb26994fe7a8ef91556d0a9042 P 61447010247c4936bc9a9fcb2af3d202: Bootstrap starting.
I20260924 04:08:58.776388 8438 raft_consensus.cc:348] T ccb307eb26994fe7a8ef91556d0a9042 P 14aa376377e34cae9a387069ec429d4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc944248f10b48bc98c503b0f733a0e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45551 } } peers { permanent_uuid: "14aa376377e34cae9a387069ec429d4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42625 } } peers { permanent_uuid: "61447010247c4936bc9a9fcb2af3d202" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45391 } }
I20260924 04:08:58.776693 8438 raft_consensus.cc:374] T ccb307eb26994fe7a8ef91556d0a9042 P 14aa376377e34cae9a387069ec429d4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:08:58.776772 8438 raft_consensus.cc:729] T ccb307eb26994fe7a8ef91556d0a9042 P 14aa376377e34cae9a387069ec429d4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14aa376377e34cae9a387069ec429d4a, State: Initialized, Role: FOLLOWER
I20260924 04:08:58.777128 8437 raft_consensus.cc:348] T ccb307eb26994fe7a8ef91556d0a9042 P cc944248f10b48bc98c503b0f733a0e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc944248f10b48bc98c503b0f733a0e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45551 } } peers { permanent_uuid: "14aa376377e34cae9a387069ec429d4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42625 } } peers { permanent_uuid: "61447010247c4936bc9a9fcb2af3d202" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45391 } }
I20260924 04:08:58.777267 8438 consensus_queue.cc:260] T ccb307eb26994fe7a8ef91556d0a9042 P 14aa376377e34cae9a387069ec429d4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc944248f10b48bc98c503b0f733a0e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45551 } } peers { permanent_uuid: "14aa376377e34cae9a387069ec429d4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42625 } } peers { permanent_uuid: "61447010247c4936bc9a9fcb2af3d202" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45391 } }
I20260924 04:08:58.777346 8437 raft_consensus.cc:374] T ccb307eb26994fe7a8ef91556d0a9042 P cc944248f10b48bc98c503b0f733a0e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:08:58.777396 8437 raft_consensus.cc:729] T ccb307eb26994fe7a8ef91556d0a9042 P cc944248f10b48bc98c503b0f733a0e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc944248f10b48bc98c503b0f733a0e9, State: Initialized, Role: FOLLOWER
I20260924 04:08:58.777843 8437 consensus_queue.cc:260] T ccb307eb26994fe7a8ef91556d0a9042 P cc944248f10b48bc98c503b0f733a0e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc944248f10b48bc98c503b0f733a0e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45551 } } peers { permanent_uuid: "14aa376377e34cae9a387069ec429d4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42625 } } peers { permanent_uuid: "61447010247c4936bc9a9fcb2af3d202" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45391 } }
I20260924 04:08:58.778021 8409 heartbeater.cc:503] Master 127.0.98.62:32951 was elected leader, sending a full tablet report...
I20260924 04:08:58.778342 8438 ts_tablet_manager.cc:1428] T ccb307eb26994fe7a8ef91556d0a9042 P 14aa376377e34cae9a387069ec429d4a: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260924 04:08:58.778797 8437 ts_tablet_manager.cc:1428] T ccb307eb26994fe7a8ef91556d0a9042 P cc944248f10b48bc98c503b0f733a0e9: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260924 04:08:58.779176 8129 heartbeater.cc:503] Master 127.0.98.62:32951 was elected leader, sending a full tablet report...
I20260924 04:08:58.779640 8441 tablet_bootstrap.cc:654] T ccb307eb26994fe7a8ef91556d0a9042 P 61447010247c4936bc9a9fcb2af3d202: Neither blocks nor log segments found. Creating new log.
I20260924 04:08:58.780392 8441 log.cc:824] T ccb307eb26994fe7a8ef91556d0a9042 P 61447010247c4936bc9a9fcb2af3d202: Log is configured to *not* fsync() on all Append() calls
I20260924 04:08:58.783044 8441 tablet_bootstrap.cc:492] T ccb307eb26994fe7a8ef91556d0a9042 P 61447010247c4936bc9a9fcb2af3d202: No bootstrap required, opened a new log
I20260924 04:08:58.783244 8441 ts_tablet_manager.cc:1397] T ccb307eb26994fe7a8ef91556d0a9042 P 61447010247c4936bc9a9fcb2af3d202: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20260924 04:08:58.786149 8441 raft_consensus.cc:348] T ccb307eb26994fe7a8ef91556d0a9042 P 61447010247c4936bc9a9fcb2af3d202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc944248f10b48bc98c503b0f733a0e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45551 } } peers { permanent_uuid: "14aa376377e34cae9a387069ec429d4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42625 } } peers { permanent_uuid: "61447010247c4936bc9a9fcb2af3d202" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45391 } }
I20260924 04:08:58.786414 8441 raft_consensus.cc:374] T ccb307eb26994fe7a8ef91556d0a9042 P 61447010247c4936bc9a9fcb2af3d202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:08:58.786470 8441 raft_consensus.cc:729] T ccb307eb26994fe7a8ef91556d0a9042 P 61447010247c4936bc9a9fcb2af3d202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61447010247c4936bc9a9fcb2af3d202, State: Initialized, Role: FOLLOWER
I20260924 04:08:58.786903 8441 consensus_queue.cc:260] T ccb307eb26994fe7a8ef91556d0a9042 P 61447010247c4936bc9a9fcb2af3d202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc944248f10b48bc98c503b0f733a0e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45551 } } peers { permanent_uuid: "14aa376377e34cae9a387069ec429d4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42625 } } peers { permanent_uuid: "61447010247c4936bc9a9fcb2af3d202" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45391 } }
I20260924 04:08:58.787591 8269 heartbeater.cc:503] Master 127.0.98.62:32951 was elected leader, sending a full tablet report...
I20260924 04:08:58.787921 8441 ts_tablet_manager.cc:1428] T ccb307eb26994fe7a8ef91556d0a9042 P 61447010247c4936bc9a9fcb2af3d202: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20260924 04:08:58.788415 8442 raft_consensus.cc:482] T ccb307eb26994fe7a8ef91556d0a9042 P 14aa376377e34cae9a387069ec429d4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:08:58.788587 8442 raft_consensus.cc:504] T ccb307eb26994fe7a8ef91556d0a9042 P 14aa376377e34cae9a387069ec429d4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc944248f10b48bc98c503b0f733a0e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45551 } } peers { permanent_uuid: "14aa376377e34cae9a387069ec429d4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42625 } } peers { permanent_uuid: "61447010247c4936bc9a9fcb2af3d202" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45391 } }
I20260924 04:08:58.789681 8442 leader_election.cc:288] T ccb307eb26994fe7a8ef91556d0a9042 P 14aa376377e34cae9a387069ec429d4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc944248f10b48bc98c503b0f733a0e9 (127.0.98.1:45551), 61447010247c4936bc9a9fcb2af3d202 (127.0.98.2:45391)
I20260924 04:08:58.794219 8302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:58.789927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45551 (local address 127.0.98.3:53741)
0924 04:08:58.790046 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:58.790061 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:58.790143 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:58.790421 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:58.790424 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:58.790444 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:08:58.790571 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:58.790578 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:58.791708 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:58.791711 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:58.793058 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:58.793069 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:58.793770 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:58.793772 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:58.793892 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:58.793917 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 04:08:58.793997 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 04:08:58.794084 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260924 04:08:58.794252 8435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:58.789989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53741 (local address 127.0.98.1:45551)
0924 04:08:58.790139 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:58.790146 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:58.790161 (+ 15us) server_negotiation.cc:407] Connection header received
0924 04:08:58.790250 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:58.790255 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:58.790313 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:58.790398 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:08:58.790859 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:58.791604 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:58.793174 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:58.793663 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:58.793767 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:58.793844 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:58.794017 (+ 173us) server_negotiation.cc:299] Negotiation successful
0924 04:08:58.794108 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 04:08:58.794677 8446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:58.790209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45391 (local address 127.0.98.3:55143)
0924 04:08:58.790496 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:58.790508 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:58.790582 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:58.790793 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:58.790795 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:58.790811 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:08:58.790935 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:58.790944 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:58.791843 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:58.791845 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:58.793406 (+ 1561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:58.793413 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:58.794238 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:58.794243 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:58.794342 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:58.794369 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 04:08:58.794491 (+ 122us) client_negotiation.cc:240] Negotiation successful
0924 04:08:58.794550 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":132,"threads_started":1}
I20260924 04:08:58.794709 8436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:58.790314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55143 (local address 127.0.98.2:45391)
0924 04:08:58.790575 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:58.790580 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:58.790593 (+ 13us) server_negotiation.cc:407] Connection header received
0924 04:08:58.790657 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:58.790660 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:58.790706 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:58.790773 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:08:58.791072 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:58.791753 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:58.793509 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:58.794126 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:58.794223 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:58.794280 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:58.794528 (+ 248us) server_negotiation.cc:299] Negotiation successful
0924 04:08:58.794609 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173}
I20260924 04:08:58.794821 8083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccb307eb26994fe7a8ef91556d0a9042" candidate_uuid: "14aa376377e34cae9a387069ec429d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc944248f10b48bc98c503b0f733a0e9" is_pre_election: true
I20260924 04:08:58.795154 8083 raft_consensus.cc:2454] T ccb307eb26994fe7a8ef91556d0a9042 P cc944248f10b48bc98c503b0f733a0e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14aa376377e34cae9a387069ec429d4a in term 0.
I20260924 04:08:58.795324 8223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccb307eb26994fe7a8ef91556d0a9042" candidate_uuid: "14aa376377e34cae9a387069ec429d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61447010247c4936bc9a9fcb2af3d202" is_pre_election: true
I20260924 04:08:58.795563 8223 raft_consensus.cc:2454] T ccb307eb26994fe7a8ef91556d0a9042 P 61447010247c4936bc9a9fcb2af3d202 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14aa376377e34cae9a387069ec429d4a in term 0.
I20260924 04:08:58.795650 8295 leader_election.cc:302] T ccb307eb26994fe7a8ef91556d0a9042 P 14aa376377e34cae9a387069ec429d4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14aa376377e34cae9a387069ec429d4a, cc944248f10b48bc98c503b0f733a0e9; no voters:
I20260924 04:08:58.795938 8442 raft_consensus.cc:2781] T ccb307eb26994fe7a8ef91556d0a9042 P 14aa376377e34cae9a387069ec429d4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:08:58.796022 8442 raft_consensus.cc:482] T ccb307eb26994fe7a8ef91556d0a9042 P 14aa376377e34cae9a387069ec429d4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:08:58.796065 8442 raft_consensus.cc:3037] T ccb307eb26994fe7a8ef91556d0a9042 P 14aa376377e34cae9a387069ec429d4a [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:08:58.797224 8442 raft_consensus.cc:504] T ccb307eb26994fe7a8ef91556d0a9042 P 14aa376377e34cae9a387069ec429d4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc944248f10b48bc98c503b0f733a0e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45551 } } peers { permanent_uuid: "14aa376377e34cae9a387069ec429d4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42625 } } peers { permanent_uuid: "61447010247c4936bc9a9fcb2af3d202" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45391 } }
I20260924 04:08:58.797609 8442 leader_election.cc:288] T ccb307eb26994fe7a8ef91556d0a9042 P 14aa376377e34cae9a387069ec429d4a [CANDIDATE]: Term 1 election: Requested vote from peers cc944248f10b48bc98c503b0f733a0e9 (127.0.98.1:45551), 61447010247c4936bc9a9fcb2af3d202 (127.0.98.2:45391)
I20260924 04:08:58.798039 8083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccb307eb26994fe7a8ef91556d0a9042" candidate_uuid: "14aa376377e34cae9a387069ec429d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc944248f10b48bc98c503b0f733a0e9"
I20260924 04:08:58.798133 8223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccb307eb26994fe7a8ef91556d0a9042" candidate_uuid: "14aa376377e34cae9a387069ec429d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61447010247c4936bc9a9fcb2af3d202"
I20260924 04:08:58.798147 8083 raft_consensus.cc:3037] T ccb307eb26994fe7a8ef91556d0a9042 P cc944248f10b48bc98c503b0f733a0e9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:08:58.798254 8223 raft_consensus.cc:3037] T ccb307eb26994fe7a8ef91556d0a9042 P 61447010247c4936bc9a9fcb2af3d202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:08:58.799188 8083 raft_consensus.cc:2454] T ccb307eb26994fe7a8ef91556d0a9042 P cc944248f10b48bc98c503b0f733a0e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14aa376377e34cae9a387069ec429d4a in term 1.
I20260924 04:08:58.799368 8223 raft_consensus.cc:2454] T ccb307eb26994fe7a8ef91556d0a9042 P 61447010247c4936bc9a9fcb2af3d202 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14aa376377e34cae9a387069ec429d4a in term 1.
I20260924 04:08:58.799477 8295 leader_election.cc:302] T ccb307eb26994fe7a8ef91556d0a9042 P 14aa376377e34cae9a387069ec429d4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14aa376377e34cae9a387069ec429d4a, cc944248f10b48bc98c503b0f733a0e9; no voters:
I20260924 04:08:58.799624 8442 raft_consensus.cc:2781] T ccb307eb26994fe7a8ef91556d0a9042 P 14aa376377e34cae9a387069ec429d4a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:08:58.799851 8442 raft_consensus.cc:686] T ccb307eb26994fe7a8ef91556d0a9042 P 14aa376377e34cae9a387069ec429d4a [term 1 LEADER]: Becoming Leader. State: Replica: 14aa376377e34cae9a387069ec429d4a, State: Running, Role: LEADER
I20260924 04:08:58.800206 8442 consensus_queue.cc:237] T ccb307eb26994fe7a8ef91556d0a9042 P 14aa376377e34cae9a387069ec429d4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc944248f10b48bc98c503b0f733a0e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45551 } } peers { permanent_uuid: "14aa376377e34cae9a387069ec429d4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42625 } } peers { permanent_uuid: "61447010247c4936bc9a9fcb2af3d202" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45391 } }
I20260924 04:08:58.803344 7926 catalog_manager.cc:5495] T ccb307eb26994fe7a8ef91556d0a9042 P 14aa376377e34cae9a387069ec429d4a reported cstate change: term changed from 0 to 1, leader changed from <none> to 14aa376377e34cae9a387069ec429d4a (127.0.98.3). New cstate: current_term: 1 leader_uuid: "14aa376377e34cae9a387069ec429d4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc944248f10b48bc98c503b0f733a0e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14aa376377e34cae9a387069ec429d4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61447010247c4936bc9a9fcb2af3d202" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45391 } health_report { overall_health: UNKNOWN } } }
I20260924 04:08:58.818836 8433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:58.815452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57822 (local address 127.0.98.3:42625)
0924 04:08:58.815581 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:58.815585 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:58.815597 (+ 12us) server_negotiation.cc:407] Connection header received
0924 04:08:58.815666 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:58.815669 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:58.815722 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:58.815811 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:08:58.816111 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:58.816609 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:58.817541 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:58.817750 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:58.817827 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:58.818071 (+ 244us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:08:58.818498 (+ 427us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:08:58.818581 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:58.818675 (+ 94us) server_negotiation.cc:299] Negotiation successful
0924 04:08:58.818727 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 04:08:58.824982 8083 raft_consensus.cc:1261] T ccb307eb26994fe7a8ef91556d0a9042 P cc944248f10b48bc98c503b0f733a0e9 [term 1 FOLLOWER]: Refusing update from remote peer 14aa376377e34cae9a387069ec429d4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:08:58.825197 8223 raft_consensus.cc:1261] T ccb307eb26994fe7a8ef91556d0a9042 P 61447010247c4936bc9a9fcb2af3d202 [term 1 FOLLOWER]: Refusing update from remote peer 14aa376377e34cae9a387069ec429d4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:08:58.825701 8442 consensus_queue.cc:1035] T ccb307eb26994fe7a8ef91556d0a9042 P 14aa376377e34cae9a387069ec429d4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc944248f10b48bc98c503b0f733a0e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:08:58.825871 8447 consensus_queue.cc:1035] T ccb307eb26994fe7a8ef91556d0a9042 P 14aa376377e34cae9a387069ec429d4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "61447010247c4936bc9a9fcb2af3d202" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:08:58.833985 8452 mvcc.cc:205] Tried to move back new op lower bound from 7332753157419302912 to 7332753157328723968. Current Snapshot: MvccSnapshot[applied={T|T < 7332753157419302912}]
I20260924 04:08:58.842547 7926 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35020:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:08:58.842713 7926 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35020:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:08:58.845165 7926 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 37584f0cfe7049d291a195adfd8d7aaf: Sending DeleteTablet for 3 replicas of tablet ccb307eb26994fe7a8ef91556d0a9042
I20260924 04:08:58.846063 8343 tablet_service.cc:1514] Processing DeleteTablet for tablet ccb307eb26994fe7a8ef91556d0a9042 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:08:58 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57816
I20260924 04:08:58.846131 8203 tablet_service.cc:1514] Processing DeleteTablet for tablet ccb307eb26994fe7a8ef91556d0a9042 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:08:58 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35930
I20260924 04:08:58.846131 8063 tablet_service.cc:1514] Processing DeleteTablet for tablet ccb307eb26994fe7a8ef91556d0a9042 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:08:58 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38806
I20260924 04:08:58.846771 8459 tablet_replica.cc:330] T ccb307eb26994fe7a8ef91556d0a9042 P 14aa376377e34cae9a387069ec429d4a: stopping tablet replica
I20260924 04:08:58.846825 8460 tablet_replica.cc:330] T ccb307eb26994fe7a8ef91556d0a9042 P 61447010247c4936bc9a9fcb2af3d202: stopping tablet replica
I20260924 04:08:58.847155 8459 raft_consensus.cc:2229] T ccb307eb26994fe7a8ef91556d0a9042 P 14aa376377e34cae9a387069ec429d4a [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:08:58.847348 8460 raft_consensus.cc:2229] T ccb307eb26994fe7a8ef91556d0a9042 P 61447010247c4936bc9a9fcb2af3d202 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:08:58.847491 7926 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35020:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 04:08:58.847620 8459 raft_consensus.cc:2258] T ccb307eb26994fe7a8ef91556d0a9042 P 14aa376377e34cae9a387069ec429d4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:08:58.847697 8460 raft_consensus.cc:2258] T ccb307eb26994fe7a8ef91556d0a9042 P 61447010247c4936bc9a9fcb2af3d202 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260924 04:08:58.847942 7926 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:08:58.849249 8460 ts_tablet_manager.cc:1905] T ccb307eb26994fe7a8ef91556d0a9042 P 61447010247c4936bc9a9fcb2af3d202: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:08:58.849606 8459 ts_tablet_manager.cc:1905] T ccb307eb26994fe7a8ef91556d0a9042 P 14aa376377e34cae9a387069ec429d4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:08:58.852356 8459 ts_tablet_manager.cc:1918] T ccb307eb26994fe7a8ef91556d0a9042 P 14aa376377e34cae9a387069ec429d4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:08:58.852456 8459 log.cc:1196] T ccb307eb26994fe7a8ef91556d0a9042 P 14aa376377e34cae9a387069ec429d4a: Deleting WAL directory at /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/minicluster-data/ts-2/wal/wals/ccb307eb26994fe7a8ef91556d0a9042
I20260924 04:08:58.852835 8459 ts_tablet_manager.cc:1939] T ccb307eb26994fe7a8ef91556d0a9042 P 14aa376377e34cae9a387069ec429d4a: Deleting consensus metadata
I20260924 04:08:58.853341 8203 tablet_service.cc:1467] Processing CreateTablet for tablet c4f1d65d6daa4eeaae734cadcf7d322c (DEFAULT_TABLE table=test-table [id=b207630f9223449d8456bf2fddd51310]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:08:58.853920 8343 tablet_service.cc:1467] Processing CreateTablet for tablet c4f1d65d6daa4eeaae734cadcf7d322c (DEFAULT_TABLE table=test-table [id=b207630f9223449d8456bf2fddd51310]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:08:58.854074 7910 catalog_manager.cc:4841] TS 14aa376377e34cae9a387069ec429d4a (127.0.98.3:42625): tablet ccb307eb26994fe7a8ef91556d0a9042 (table test-table [id=db3405d46f5f4cccb360fd94f3d62ff8]) successfully deleted
I20260924 04:08:58.854100 8063 tablet_service.cc:1467] Processing CreateTablet for tablet c4f1d65d6daa4eeaae734cadcf7d322c (DEFAULT_TABLE table=test-table [id=b207630f9223449d8456bf2fddd51310]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:08:58.854167 8343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4f1d65d6daa4eeaae734cadcf7d322c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:08:58.854354 8063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4f1d65d6daa4eeaae734cadcf7d322c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:08:58.854368 8461 tablet_replica.cc:330] T ccb307eb26994fe7a8ef91556d0a9042 P cc944248f10b48bc98c503b0f733a0e9: stopping tablet replica
I20260924 04:08:58.854725 8461 raft_consensus.cc:2229] T ccb307eb26994fe7a8ef91556d0a9042 P cc944248f10b48bc98c503b0f733a0e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:08:58.855043 8461 raft_consensus.cc:2258] T ccb307eb26994fe7a8ef91556d0a9042 P cc944248f10b48bc98c503b0f733a0e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:08:58.856375 8438 tablet_bootstrap.cc:492] T c4f1d65d6daa4eeaae734cadcf7d322c P 14aa376377e34cae9a387069ec429d4a: Bootstrap starting.
I20260924 04:08:58.856761 8437 tablet_bootstrap.cc:492] T c4f1d65d6daa4eeaae734cadcf7d322c P cc944248f10b48bc98c503b0f733a0e9: Bootstrap starting.
I20260924 04:08:58.857375 8438 tablet_bootstrap.cc:654] T c4f1d65d6daa4eeaae734cadcf7d322c P 14aa376377e34cae9a387069ec429d4a: Neither blocks nor log segments found. Creating new log.
I20260924 04:08:58.857730 8437 tablet_bootstrap.cc:654] T c4f1d65d6daa4eeaae734cadcf7d322c P cc944248f10b48bc98c503b0f733a0e9: Neither blocks nor log segments found. Creating new log.
I20260924 04:08:58.857757 8203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4f1d65d6daa4eeaae734cadcf7d322c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:08:58.858466 8461 ts_tablet_manager.cc:1905] T ccb307eb26994fe7a8ef91556d0a9042 P cc944248f10b48bc98c503b0f733a0e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:08:58.858996 8437 tablet_bootstrap.cc:492] T c4f1d65d6daa4eeaae734cadcf7d322c P cc944248f10b48bc98c503b0f733a0e9: No bootstrap required, opened a new log
I20260924 04:08:58.859073 8437 ts_tablet_manager.cc:1397] T c4f1d65d6daa4eeaae734cadcf7d322c P cc944248f10b48bc98c503b0f733a0e9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 04:08:58.859542 8437 raft_consensus.cc:348] T c4f1d65d6daa4eeaae734cadcf7d322c P cc944248f10b48bc98c503b0f733a0e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc944248f10b48bc98c503b0f733a0e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45551 } } peers { permanent_uuid: "61447010247c4936bc9a9fcb2af3d202" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45391 } } peers { permanent_uuid: "14aa376377e34cae9a387069ec429d4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42625 } }
I20260924 04:08:58.859652 8437 raft_consensus.cc:374] T c4f1d65d6daa4eeaae734cadcf7d322c P cc944248f10b48bc98c503b0f733a0e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:08:58.859683 8437 raft_consensus.cc:729] T c4f1d65d6daa4eeaae734cadcf7d322c P cc944248f10b48bc98c503b0f733a0e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc944248f10b48bc98c503b0f733a0e9, State: Initialized, Role: FOLLOWER
I20260924 04:08:58.859831 8437 consensus_queue.cc:260] T c4f1d65d6daa4eeaae734cadcf7d322c P cc944248f10b48bc98c503b0f733a0e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc944248f10b48bc98c503b0f733a0e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45551 } } peers { permanent_uuid: "61447010247c4936bc9a9fcb2af3d202" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45391 } } peers { permanent_uuid: "14aa376377e34cae9a387069ec429d4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42625 } }
I20260924 04:08:58.860134 8437 ts_tablet_manager.cc:1428] T c4f1d65d6daa4eeaae734cadcf7d322c P cc944248f10b48bc98c503b0f733a0e9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:08:58.860350 8441 tablet_bootstrap.cc:492] T c4f1d65d6daa4eeaae734cadcf7d322c P 61447010247c4936bc9a9fcb2af3d202: Bootstrap starting.
I20260924 04:08:58.861001 8461 ts_tablet_manager.cc:1918] T ccb307eb26994fe7a8ef91556d0a9042 P cc944248f10b48bc98c503b0f733a0e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:08:58.861100 8461 log.cc:1196] T ccb307eb26994fe7a8ef91556d0a9042 P cc944248f10b48bc98c503b0f733a0e9: Deleting WAL directory at /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/minicluster-data/ts-0/wal/wals/ccb307eb26994fe7a8ef91556d0a9042
I20260924 04:08:58.861447 8441 tablet_bootstrap.cc:654] T c4f1d65d6daa4eeaae734cadcf7d322c P 61447010247c4936bc9a9fcb2af3d202: Neither blocks nor log segments found. Creating new log.
I20260924 04:08:58.861784 8461 ts_tablet_manager.cc:1939] T ccb307eb26994fe7a8ef91556d0a9042 P cc944248f10b48bc98c503b0f733a0e9: Deleting consensus metadata
I20260924 04:08:58.862998 7911 catalog_manager.cc:4841] TS cc944248f10b48bc98c503b0f733a0e9 (127.0.98.1:45551): tablet ccb307eb26994fe7a8ef91556d0a9042 (table test-table [id=db3405d46f5f4cccb360fd94f3d62ff8]) successfully deleted
I20260924 04:08:58.863718 8438 tablet_bootstrap.cc:492] T c4f1d65d6daa4eeaae734cadcf7d322c P 14aa376377e34cae9a387069ec429d4a: No bootstrap required, opened a new log
I20260924 04:08:58.863803 8438 ts_tablet_manager.cc:1397] T c4f1d65d6daa4eeaae734cadcf7d322c P 14aa376377e34cae9a387069ec429d4a: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260924 04:08:58.863999 8460 ts_tablet_manager.cc:1918] T ccb307eb26994fe7a8ef91556d0a9042 P 61447010247c4936bc9a9fcb2af3d202: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:08:58.864069 8460 log.cc:1196] T ccb307eb26994fe7a8ef91556d0a9042 P 61447010247c4936bc9a9fcb2af3d202: Deleting WAL directory at /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/minicluster-data/ts-1/wal/wals/ccb307eb26994fe7a8ef91556d0a9042
I20260924 04:08:58.864283 8460 ts_tablet_manager.cc:1939] T ccb307eb26994fe7a8ef91556d0a9042 P 61447010247c4936bc9a9fcb2af3d202: Deleting consensus metadata
I20260924 04:08:58.864234 8438 raft_consensus.cc:348] T c4f1d65d6daa4eeaae734cadcf7d322c P 14aa376377e34cae9a387069ec429d4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc944248f10b48bc98c503b0f733a0e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45551 } } peers { permanent_uuid: "61447010247c4936bc9a9fcb2af3d202" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45391 } } peers { permanent_uuid: "14aa376377e34cae9a387069ec429d4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42625 } }
I20260924 04:08:58.864337 8438 raft_consensus.cc:374] T c4f1d65d6daa4eeaae734cadcf7d322c P 14aa376377e34cae9a387069ec429d4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:08:58.864367 8438 raft_consensus.cc:729] T c4f1d65d6daa4eeaae734cadcf7d322c P 14aa376377e34cae9a387069ec429d4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14aa376377e34cae9a387069ec429d4a, State: Initialized, Role: FOLLOWER
I20260924 04:08:58.864494 8438 consensus_queue.cc:260] T c4f1d65d6daa4eeaae734cadcf7d322c P 14aa376377e34cae9a387069ec429d4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc944248f10b48bc98c503b0f733a0e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45551 } } peers { permanent_uuid: "61447010247c4936bc9a9fcb2af3d202" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45391 } } peers { permanent_uuid: "14aa376377e34cae9a387069ec429d4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42625 } }
I20260924 04:08:58.864758 8438 ts_tablet_manager.cc:1428] T c4f1d65d6daa4eeaae734cadcf7d322c P 14aa376377e34cae9a387069ec429d4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:08:58.865800 7911 catalog_manager.cc:4841] TS 61447010247c4936bc9a9fcb2af3d202 (127.0.98.2:45391): tablet ccb307eb26994fe7a8ef91556d0a9042 (table test-table [id=db3405d46f5f4cccb360fd94f3d62ff8]) successfully deleted
I20260924 04:08:58.866246 8441 tablet_bootstrap.cc:492] T c4f1d65d6daa4eeaae734cadcf7d322c P 61447010247c4936bc9a9fcb2af3d202: No bootstrap required, opened a new log
I20260924 04:08:58.866345 8441 ts_tablet_manager.cc:1397] T c4f1d65d6daa4eeaae734cadcf7d322c P 61447010247c4936bc9a9fcb2af3d202: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260924 04:08:58.866748 8441 raft_consensus.cc:348] T c4f1d65d6daa4eeaae734cadcf7d322c P 61447010247c4936bc9a9fcb2af3d202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc944248f10b48bc98c503b0f733a0e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45551 } } peers { permanent_uuid: "61447010247c4936bc9a9fcb2af3d202" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45391 } } peers { permanent_uuid: "14aa376377e34cae9a387069ec429d4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42625 } }
I20260924 04:08:58.866864 8441 raft_consensus.cc:374] T c4f1d65d6daa4eeaae734cadcf7d322c P 61447010247c4936bc9a9fcb2af3d202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:08:58.866904 8441 raft_consensus.cc:729] T c4f1d65d6daa4eeaae734cadcf7d322c P 61447010247c4936bc9a9fcb2af3d202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61447010247c4936bc9a9fcb2af3d202, State: Initialized, Role: FOLLOWER
I20260924 04:08:58.867043 8441 consensus_queue.cc:260] T c4f1d65d6daa4eeaae734cadcf7d322c P 61447010247c4936bc9a9fcb2af3d202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc944248f10b48bc98c503b0f733a0e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45551 } } peers { permanent_uuid: "61447010247c4936bc9a9fcb2af3d202" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45391 } } peers { permanent_uuid: "14aa376377e34cae9a387069ec429d4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42625 } }
I20260924 04:08:58.867311 8441 ts_tablet_manager.cc:1428] T c4f1d65d6daa4eeaae734cadcf7d322c P 61447010247c4936bc9a9fcb2af3d202: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:08:58.894170 8447 raft_consensus.cc:482] T c4f1d65d6daa4eeaae734cadcf7d322c P 14aa376377e34cae9a387069ec429d4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:08:58.894321 8447 raft_consensus.cc:504] T c4f1d65d6daa4eeaae734cadcf7d322c P 14aa376377e34cae9a387069ec429d4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc944248f10b48bc98c503b0f733a0e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45551 } } peers { permanent_uuid: "61447010247c4936bc9a9fcb2af3d202" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45391 } } peers { permanent_uuid: "14aa376377e34cae9a387069ec429d4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42625 } }
I20260924 04:08:58.894757 8447 leader_election.cc:288] T c4f1d65d6daa4eeaae734cadcf7d322c P 14aa376377e34cae9a387069ec429d4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc944248f10b48bc98c503b0f733a0e9 (127.0.98.1:45551), 61447010247c4936bc9a9fcb2af3d202 (127.0.98.2:45391)
I20260924 04:08:58.895233 8223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f1d65d6daa4eeaae734cadcf7d322c" candidate_uuid: "14aa376377e34cae9a387069ec429d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61447010247c4936bc9a9fcb2af3d202" is_pre_election: true
I20260924 04:08:58.895207 8083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f1d65d6daa4eeaae734cadcf7d322c" candidate_uuid: "14aa376377e34cae9a387069ec429d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc944248f10b48bc98c503b0f733a0e9" is_pre_election: true
I20260924 04:08:58.895373 8083 raft_consensus.cc:2454] T c4f1d65d6daa4eeaae734cadcf7d322c P cc944248f10b48bc98c503b0f733a0e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14aa376377e34cae9a387069ec429d4a in term 0.
I20260924 04:08:58.895372 8223 raft_consensus.cc:2454] T c4f1d65d6daa4eeaae734cadcf7d322c P 61447010247c4936bc9a9fcb2af3d202 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14aa376377e34cae9a387069ec429d4a in term 0.
I20260924 04:08:58.895670 8295 leader_election.cc:302] T c4f1d65d6daa4eeaae734cadcf7d322c P 14aa376377e34cae9a387069ec429d4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14aa376377e34cae9a387069ec429d4a, 61447010247c4936bc9a9fcb2af3d202; no voters:
I20260924 04:08:58.895957 8447 raft_consensus.cc:2781] T c4f1d65d6daa4eeaae734cadcf7d322c P 14aa376377e34cae9a387069ec429d4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:08:58.896016 8447 raft_consensus.cc:482] T c4f1d65d6daa4eeaae734cadcf7d322c P 14aa376377e34cae9a387069ec429d4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:08:58.896054 8447 raft_consensus.cc:3037] T c4f1d65d6daa4eeaae734cadcf7d322c P 14aa376377e34cae9a387069ec429d4a [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:08:58.897370 8447 raft_consensus.cc:504] T c4f1d65d6daa4eeaae734cadcf7d322c P 14aa376377e34cae9a387069ec429d4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc944248f10b48bc98c503b0f733a0e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45551 } } peers { permanent_uuid: "61447010247c4936bc9a9fcb2af3d202" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45391 } } peers { permanent_uuid: "14aa376377e34cae9a387069ec429d4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42625 } }
I20260924 04:08:58.897703 8447 leader_election.cc:288] T c4f1d65d6daa4eeaae734cadcf7d322c P 14aa376377e34cae9a387069ec429d4a [CANDIDATE]: Term 1 election: Requested vote from peers cc944248f10b48bc98c503b0f733a0e9 (127.0.98.1:45551), 61447010247c4936bc9a9fcb2af3d202 (127.0.98.2:45391)
I20260924 04:08:58.898136 8083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f1d65d6daa4eeaae734cadcf7d322c" candidate_uuid: "14aa376377e34cae9a387069ec429d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc944248f10b48bc98c503b0f733a0e9"
I20260924 04:08:58.898232 8223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f1d65d6daa4eeaae734cadcf7d322c" candidate_uuid: "14aa376377e34cae9a387069ec429d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61447010247c4936bc9a9fcb2af3d202"
I20260924 04:08:58.898283 8083 raft_consensus.cc:3037] T c4f1d65d6daa4eeaae734cadcf7d322c P cc944248f10b48bc98c503b0f733a0e9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:08:58.898370 8223 raft_consensus.cc:3037] T c4f1d65d6daa4eeaae734cadcf7d322c P 61447010247c4936bc9a9fcb2af3d202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:08:58.899451 8223 raft_consensus.cc:2454] T c4f1d65d6daa4eeaae734cadcf7d322c P 61447010247c4936bc9a9fcb2af3d202 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14aa376377e34cae9a387069ec429d4a in term 1.
I20260924 04:08:58.899452 8083 raft_consensus.cc:2454] T c4f1d65d6daa4eeaae734cadcf7d322c P cc944248f10b48bc98c503b0f733a0e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14aa376377e34cae9a387069ec429d4a in term 1.
I20260924 04:08:58.899715 8295 leader_election.cc:302] T c4f1d65d6daa4eeaae734cadcf7d322c P 14aa376377e34cae9a387069ec429d4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14aa376377e34cae9a387069ec429d4a, cc944248f10b48bc98c503b0f733a0e9; no voters:
I20260924 04:08:58.899962 8447 raft_consensus.cc:2781] T c4f1d65d6daa4eeaae734cadcf7d322c P 14aa376377e34cae9a387069ec429d4a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:08:58.900051 8447 raft_consensus.cc:686] T c4f1d65d6daa4eeaae734cadcf7d322c P 14aa376377e34cae9a387069ec429d4a [term 1 LEADER]: Becoming Leader. State: Replica: 14aa376377e34cae9a387069ec429d4a, State: Running, Role: LEADER
I20260924 04:08:58.900172 8447 consensus_queue.cc:237] T c4f1d65d6daa4eeaae734cadcf7d322c P 14aa376377e34cae9a387069ec429d4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc944248f10b48bc98c503b0f733a0e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45551 } } peers { permanent_uuid: "61447010247c4936bc9a9fcb2af3d202" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45391 } } peers { permanent_uuid: "14aa376377e34cae9a387069ec429d4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42625 } }
I20260924 04:08:58.901919 7925 catalog_manager.cc:5495] T c4f1d65d6daa4eeaae734cadcf7d322c P 14aa376377e34cae9a387069ec429d4a reported cstate change: term changed from 0 to 1, leader changed from <none> to 14aa376377e34cae9a387069ec429d4a (127.0.98.3). New cstate: current_term: 1 leader_uuid: "14aa376377e34cae9a387069ec429d4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc944248f10b48bc98c503b0f733a0e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "61447010247c4936bc9a9fcb2af3d202" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14aa376377e34cae9a387069ec429d4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42625 } health_report { overall_health: HEALTHY } } }
Sep 24 04:08:58 dist-test-slave-00ld krb5kdc[7881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222937, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
W20260924 04:08:58.939865 8410 tablet.cc:2369] T c4f1d65d6daa4eeaae734cadcf7d322c P 14aa376377e34cae9a387069ec429d4a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:08:58.943295 7975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:58.934467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35032 (local address 127.0.98.62:32951)
0924 04:08:58.934596 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:58.934600 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:58.934782 (+ 182us) server_negotiation.cc:407] Connection header received
0924 04:08:58.934906 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:58.934908 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:58.934947 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:58.935038 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:58.935836 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:58.936354 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:58.937523 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:58.937763 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:58.937890 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:58.940110 (+ 2220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:58.940113 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:58.940115 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:58.940134 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:58.941627 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:58.942069 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:58.942074 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:58.942076 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:58.942146 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:58.942404 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:58.942407 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:58.942409 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:58.942579 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:58.942681 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:58.942965 (+ 284us) server_negotiation.cc:299] Negotiation successful
0924 04:08:58.943171 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":102,"server-negotiator.queue_time_us":53}
I20260924 04:08:58.943306 8471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:58.934400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32951 (local address 127.0.0.1:35032)
0924 04:08:58.934684 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:58.934700 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:58.934821 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:58.935083 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:58.935085 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:58.935608 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:08:58.935721 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:58.935728 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:58.936451 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:58.936454 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:58.937417 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:58.937426 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:58.937892 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:58.937895 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:58.937978 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:58.938575 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:08:58.938587 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:08:58.939972 (+ 1385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:08:58.941738 (+ 1766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:58.941746 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:58.941753 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:58.941968 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:58.942231 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:08:58.942236 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:08:58.942238 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:08:58.942318 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:08:58.942680 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:08:58.942687 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:08:58.942795 (+ 108us) client_negotiation.cc:769] Sending connection context
0924 04:08:58.942923 (+ 128us) client_negotiation.cc:240] Negotiation successful
0924 04:08:58.943110 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"mutex_wait_us":5,"thread_start_us":102,"threads_started":1}
I20260924 04:08:58.948336 7925 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35032:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 04:08:58.949116 7925 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:08:58.954010 8063 tablet_service.cc:1467] Processing CreateTablet for tablet 6a24fd0824ad438589d227671186deed (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a010ad6ce6a64820b8b02d8c07efd2c5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 04:08:58.954224 8343 tablet_service.cc:1467] Processing CreateTablet for tablet 6a24fd0824ad438589d227671186deed (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a010ad6ce6a64820b8b02d8c07efd2c5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 04:08:58.954433 8063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a24fd0824ad438589d227671186deed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:08:58.954588 8343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a24fd0824ad438589d227671186deed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:08:58.954496 8203 tablet_service.cc:1467] Processing CreateTablet for tablet 6a24fd0824ad438589d227671186deed (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a010ad6ce6a64820b8b02d8c07efd2c5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 04:08:58.954814 8203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a24fd0824ad438589d227671186deed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:08:58.957461 8438 tablet_bootstrap.cc:492] T 6a24fd0824ad438589d227671186deed P 14aa376377e34cae9a387069ec429d4a: Bootstrap starting.
I20260924 04:08:58.957897 8441 tablet_bootstrap.cc:492] T 6a24fd0824ad438589d227671186deed P 61447010247c4936bc9a9fcb2af3d202: Bootstrap starting.
I20260924 04:08:58.958726 8441 tablet_bootstrap.cc:654] T 6a24fd0824ad438589d227671186deed P 61447010247c4936bc9a9fcb2af3d202: Neither blocks nor log segments found. Creating new log.
I20260924 04:08:58.958775 8438 tablet_bootstrap.cc:654] T 6a24fd0824ad438589d227671186deed P 14aa376377e34cae9a387069ec429d4a: Neither blocks nor log segments found. Creating new log.
I20260924 04:08:58.959421 8437 tablet_bootstrap.cc:492] T 6a24fd0824ad438589d227671186deed P cc944248f10b48bc98c503b0f733a0e9: Bootstrap starting.
I20260924 04:08:58.960044 8441 tablet_bootstrap.cc:492] T 6a24fd0824ad438589d227671186deed P 61447010247c4936bc9a9fcb2af3d202: No bootstrap required, opened a new log
I20260924 04:08:58.960142 8441 ts_tablet_manager.cc:1397] T 6a24fd0824ad438589d227671186deed P 61447010247c4936bc9a9fcb2af3d202: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 04:08:58.960386 8437 tablet_bootstrap.cc:654] T 6a24fd0824ad438589d227671186deed P cc944248f10b48bc98c503b0f733a0e9: Neither blocks nor log segments found. Creating new log.
I20260924 04:08:58.960567 8441 raft_consensus.cc:348] T 6a24fd0824ad438589d227671186deed P 61447010247c4936bc9a9fcb2af3d202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc944248f10b48bc98c503b0f733a0e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45551 } } peers { permanent_uuid: "61447010247c4936bc9a9fcb2af3d202" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45391 } } peers { permanent_uuid: "14aa376377e34cae9a387069ec429d4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42625 } }
I20260924 04:08:58.960685 8441 raft_consensus.cc:374] T 6a24fd0824ad438589d227671186deed P 61447010247c4936bc9a9fcb2af3d202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:08:58.960743 8441 raft_consensus.cc:729] T 6a24fd0824ad438589d227671186deed P 61447010247c4936bc9a9fcb2af3d202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61447010247c4936bc9a9fcb2af3d202, State: Initialized, Role: FOLLOWER
I20260924 04:08:58.960871 8441 consensus_queue.cc:260] T 6a24fd0824ad438589d227671186deed P 61447010247c4936bc9a9fcb2af3d202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc944248f10b48bc98c503b0f733a0e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45551 } } peers { permanent_uuid: "61447010247c4936bc9a9fcb2af3d202" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45391 } } peers { permanent_uuid: "14aa376377e34cae9a387069ec429d4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42625 } }
I20260924 04:08:58.961197 8441 ts_tablet_manager.cc:1428] T 6a24fd0824ad438589d227671186deed P 61447010247c4936bc9a9fcb2af3d202: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260924 04:08:58.961571 8445 tablet_replica.cc:439] T 6a24fd0824ad438589d227671186deed P 61447010247c4936bc9a9fcb2af3d202: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc944248f10b48bc98c503b0f733a0e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45551 } } peers { permanent_uuid: "61447010247c4936bc9a9fcb2af3d202" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45391 } } peers { permanent_uuid: "14aa376377e34cae9a387069ec429d4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42625 } } }
I20260924 04:08:58.961747 8445 tablet_replica.cc:442] T 6a24fd0824ad438589d227671186deed P 61447010247c4936bc9a9fcb2af3d202: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:08:58.961949 8437 tablet_bootstrap.cc:492] T 6a24fd0824ad438589d227671186deed P cc944248f10b48bc98c503b0f733a0e9: No bootstrap required, opened a new log
I20260924 04:08:58.962035 8437 ts_tablet_manager.cc:1397] T 6a24fd0824ad438589d227671186deed P cc944248f10b48bc98c503b0f733a0e9: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260924 04:08:58.962461 8437 raft_consensus.cc:348] T 6a24fd0824ad438589d227671186deed P cc944248f10b48bc98c503b0f733a0e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc944248f10b48bc98c503b0f733a0e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45551 } } peers { permanent_uuid: "61447010247c4936bc9a9fcb2af3d202" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45391 } } peers { permanent_uuid: "14aa376377e34cae9a387069ec429d4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42625 } }
I20260924 04:08:58.962574 8437 raft_consensus.cc:374] T 6a24fd0824ad438589d227671186deed P cc944248f10b48bc98c503b0f733a0e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:08:58.962610 8437 raft_consensus.cc:729] T 6a24fd0824ad438589d227671186deed P cc944248f10b48bc98c503b0f733a0e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc944248f10b48bc98c503b0f733a0e9, State: Initialized, Role: FOLLOWER
I20260924 04:08:58.962730 8437 consensus_queue.cc:260] T 6a24fd0824ad438589d227671186deed P cc944248f10b48bc98c503b0f733a0e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc944248f10b48bc98c503b0f733a0e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45551 } } peers { permanent_uuid: "61447010247c4936bc9a9fcb2af3d202" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45391 } } peers { permanent_uuid: "14aa376377e34cae9a387069ec429d4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42625 } }
I20260924 04:08:58.962977 8437 ts_tablet_manager.cc:1428] T 6a24fd0824ad438589d227671186deed P cc944248f10b48bc98c503b0f733a0e9: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260924 04:08:58.963963 8438 tablet_bootstrap.cc:492] T 6a24fd0824ad438589d227671186deed P 14aa376377e34cae9a387069ec429d4a: No bootstrap required, opened a new log
I20260924 04:08:58.964047 8438 ts_tablet_manager.cc:1397] T 6a24fd0824ad438589d227671186deed P 14aa376377e34cae9a387069ec429d4a: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260924 04:08:58.964493 8438 raft_consensus.cc:348] T 6a24fd0824ad438589d227671186deed P 14aa376377e34cae9a387069ec429d4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc944248f10b48bc98c503b0f733a0e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45551 } } peers { permanent_uuid: "61447010247c4936bc9a9fcb2af3d202" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45391 } } peers { permanent_uuid: "14aa376377e34cae9a387069ec429d4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42625 } }
I20260924 04:08:58.964617 8438 raft_consensus.cc:374] T 6a24fd0824ad438589d227671186deed P 14aa376377e34cae9a387069ec429d4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:08:58.964651 8438 raft_consensus.cc:729] T 6a24fd0824ad438589d227671186deed P 14aa376377e34cae9a387069ec429d4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14aa376377e34cae9a387069ec429d4a, State: Initialized, Role: FOLLOWER
I20260924 04:08:58.965497 8443 tablet_replica.cc:439] T 6a24fd0824ad438589d227671186deed P cc944248f10b48bc98c503b0f733a0e9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc944248f10b48bc98c503b0f733a0e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45551 } } peers { permanent_uuid: "61447010247c4936bc9a9fcb2af3d202" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45391 } } peers { permanent_uuid: "14aa376377e34cae9a387069ec429d4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42625 } } }
I20260924 04:08:58.965643 8443 tablet_replica.cc:442] T 6a24fd0824ad438589d227671186deed P cc944248f10b48bc98c503b0f733a0e9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:08:58.965934 8438 consensus_queue.cc:260] T 6a24fd0824ad438589d227671186deed P 14aa376377e34cae9a387069ec429d4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc944248f10b48bc98c503b0f733a0e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45551 } } peers { permanent_uuid: "61447010247c4936bc9a9fcb2af3d202" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45391 } } peers { permanent_uuid: "14aa376377e34cae9a387069ec429d4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42625 } }
I20260924 04:08:58.966185 8447 tablet_replica.cc:439] T 6a24fd0824ad438589d227671186deed P 14aa376377e34cae9a387069ec429d4a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc944248f10b48bc98c503b0f733a0e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45551 } } peers { permanent_uuid: "61447010247c4936bc9a9fcb2af3d202" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45391 } } peers { permanent_uuid: "14aa376377e34cae9a387069ec429d4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42625 } } }
I20260924 04:08:58.966301 8447 tablet_replica.cc:442] T 6a24fd0824ad438589d227671186deed P 14aa376377e34cae9a387069ec429d4a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:08:58.967144 8438 ts_tablet_manager.cc:1428] T 6a24fd0824ad438589d227671186deed P 14aa376377e34cae9a387069ec429d4a: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
W20260924 04:08:58.968402 8270 tablet.cc:2369] T 6a24fd0824ad438589d227671186deed P 61447010247c4936bc9a9fcb2af3d202: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 04:08:58.997545 8130 tablet.cc:2369] T 6a24fd0824ad438589d227671186deed P cc944248f10b48bc98c503b0f733a0e9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:08:59.039474 8445 raft_consensus.cc:482] T 6a24fd0824ad438589d227671186deed P 61447010247c4936bc9a9fcb2af3d202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:08:59.039646 8445 raft_consensus.cc:504] T 6a24fd0824ad438589d227671186deed P 61447010247c4936bc9a9fcb2af3d202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc944248f10b48bc98c503b0f733a0e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45551 } } peers { permanent_uuid: "61447010247c4936bc9a9fcb2af3d202" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45391 } } peers { permanent_uuid: "14aa376377e34cae9a387069ec429d4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42625 } }
I20260924 04:08:59.040822 8445 leader_election.cc:288] T 6a24fd0824ad438589d227671186deed P 61447010247c4936bc9a9fcb2af3d202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc944248f10b48bc98c503b0f733a0e9 (127.0.98.1:45551), 14aa376377e34cae9a387069ec429d4a (127.0.98.3:42625)
I20260924 04:08:59.044564 8435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:59.041133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53031 (local address 127.0.98.1:45551)
0924 04:08:59.041265 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:59.041271 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:59.041469 (+ 198us) server_negotiation.cc:407] Connection header received
0924 04:08:59.041626 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:59.041629 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:59.041677 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:59.041754 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:08:59.041992 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:59.042488 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:59.043578 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:59.043968 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:59.044081 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:59.044141 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:59.044325 (+ 184us) server_negotiation.cc:299] Negotiation successful
0924 04:08:59.044395 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 04:08:59.044567 8433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:59.041138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48611 (local address 127.0.98.3:42625)
0924 04:08:59.041378 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:59.041384 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:59.041399 (+ 15us) server_negotiation.cc:407] Connection header received
0924 04:08:59.041554 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:59.041558 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:59.041603 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:59.041684 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:08:59.041945 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:59.042452 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:59.043612 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:59.044013 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:59.044124 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:59.044183 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:59.044354 (+ 171us) server_negotiation.cc:299] Negotiation successful
0924 04:08:59.044432 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139}
I20260924 04:08:59.044617 8475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:59.041012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42625 (local address 127.0.98.2:48611)
0924 04:08:59.041314 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:59.041331 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:59.041439 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:59.041696 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:59.041699 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:59.041714 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:08:59.041840 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:59.041845 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:59.042562 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:59.042567 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:59.043483 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:59.043493 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:59.044128 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:59.044131 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:59.044228 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:59.044248 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 04:08:59.044337 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 04:08:59.044443 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"mutex_wait_us":67,"thread_start_us":100,"threads_started":1}
I20260924 04:08:59.044881 8476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:59.041012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45551 (local address 127.0.98.2:53031)
0924 04:08:59.041386 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:59.041397 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:59.041509 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:59.041764 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:59.041766 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:59.041781 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:08:59.041896 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:59.041901 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:59.042591 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:59.042596 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:59.043444 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:59.043453 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:59.044096 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:59.044100 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:59.044199 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:59.044229 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 04:08:59.044311 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 04:08:59.044431 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"mutex_wait_us":75,"thread_start_us":78,"threads_started":1}
I20260924 04:08:59.045181 8363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a24fd0824ad438589d227671186deed" candidate_uuid: "61447010247c4936bc9a9fcb2af3d202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14aa376377e34cae9a387069ec429d4a" is_pre_election: true
I20260924 04:08:59.045336 8083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a24fd0824ad438589d227671186deed" candidate_uuid: "61447010247c4936bc9a9fcb2af3d202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc944248f10b48bc98c503b0f733a0e9" is_pre_election: true
I20260924 04:08:59.045473 8363 raft_consensus.cc:2454] T 6a24fd0824ad438589d227671186deed P 14aa376377e34cae9a387069ec429d4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61447010247c4936bc9a9fcb2af3d202 in term 0.
I20260924 04:08:59.045506 8083 raft_consensus.cc:2454] T 6a24fd0824ad438589d227671186deed P cc944248f10b48bc98c503b0f733a0e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61447010247c4936bc9a9fcb2af3d202 in term 0.
I20260924 04:08:59.045879 8155 leader_election.cc:302] T 6a24fd0824ad438589d227671186deed P 61447010247c4936bc9a9fcb2af3d202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 14aa376377e34cae9a387069ec429d4a, 61447010247c4936bc9a9fcb2af3d202, cc944248f10b48bc98c503b0f733a0e9; no voters:
I20260924 04:08:59.046186 8445 raft_consensus.cc:2781] T 6a24fd0824ad438589d227671186deed P 61447010247c4936bc9a9fcb2af3d202 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:08:59.046257 8445 raft_consensus.cc:482] T 6a24fd0824ad438589d227671186deed P 61447010247c4936bc9a9fcb2af3d202 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:08:59.046314 8445 raft_consensus.cc:3037] T 6a24fd0824ad438589d227671186deed P 61447010247c4936bc9a9fcb2af3d202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:08:59.047354 8445 raft_consensus.cc:504] T 6a24fd0824ad438589d227671186deed P 61447010247c4936bc9a9fcb2af3d202 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc944248f10b48bc98c503b0f733a0e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45551 } } peers { permanent_uuid: "61447010247c4936bc9a9fcb2af3d202" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45391 } } peers { permanent_uuid: "14aa376377e34cae9a387069ec429d4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42625 } }
I20260924 04:08:59.047703 8445 leader_election.cc:288] T 6a24fd0824ad438589d227671186deed P 61447010247c4936bc9a9fcb2af3d202 [CANDIDATE]: Term 1 election: Requested vote from peers cc944248f10b48bc98c503b0f733a0e9 (127.0.98.1:45551), 14aa376377e34cae9a387069ec429d4a (127.0.98.3:42625)
I20260924 04:08:59.048023 8083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a24fd0824ad438589d227671186deed" candidate_uuid: "61447010247c4936bc9a9fcb2af3d202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc944248f10b48bc98c503b0f733a0e9"
I20260924 04:08:59.048023 8363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a24fd0824ad438589d227671186deed" candidate_uuid: "61447010247c4936bc9a9fcb2af3d202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14aa376377e34cae9a387069ec429d4a"
I20260924 04:08:59.048143 8083 raft_consensus.cc:3037] T 6a24fd0824ad438589d227671186deed P cc944248f10b48bc98c503b0f733a0e9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:08:59.048151 8363 raft_consensus.cc:3037] T 6a24fd0824ad438589d227671186deed P 14aa376377e34cae9a387069ec429d4a [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:08:59.049264 8083 raft_consensus.cc:2454] T 6a24fd0824ad438589d227671186deed P cc944248f10b48bc98c503b0f733a0e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61447010247c4936bc9a9fcb2af3d202 in term 1.
I20260924 04:08:59.049335 8363 raft_consensus.cc:2454] T 6a24fd0824ad438589d227671186deed P 14aa376377e34cae9a387069ec429d4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61447010247c4936bc9a9fcb2af3d202 in term 1.
I20260924 04:08:59.049553 8155 leader_election.cc:302] T 6a24fd0824ad438589d227671186deed P 61447010247c4936bc9a9fcb2af3d202 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61447010247c4936bc9a9fcb2af3d202, cc944248f10b48bc98c503b0f733a0e9; no voters:
I20260924 04:08:59.049718 8445 raft_consensus.cc:2781] T 6a24fd0824ad438589d227671186deed P 61447010247c4936bc9a9fcb2af3d202 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:08:59.049885 8445 raft_consensus.cc:686] T 6a24fd0824ad438589d227671186deed P 61447010247c4936bc9a9fcb2af3d202 [term 1 LEADER]: Becoming Leader. State: Replica: 61447010247c4936bc9a9fcb2af3d202, State: Running, Role: LEADER
I20260924 04:08:59.050156 8445 consensus_queue.cc:237] T 6a24fd0824ad438589d227671186deed P 61447010247c4936bc9a9fcb2af3d202 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc944248f10b48bc98c503b0f733a0e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45551 } } peers { permanent_uuid: "61447010247c4936bc9a9fcb2af3d202" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45391 } } peers { permanent_uuid: "14aa376377e34cae9a387069ec429d4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42625 } }
I20260924 04:08:59.051342 8477 tablet_replica.cc:439] T 6a24fd0824ad438589d227671186deed P 61447010247c4936bc9a9fcb2af3d202: TxnStatusTablet state changed. Reason: New leader 61447010247c4936bc9a9fcb2af3d202. Latest consensus state: current_term: 1 leader_uuid: "61447010247c4936bc9a9fcb2af3d202" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc944248f10b48bc98c503b0f733a0e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45551 } } peers { permanent_uuid: "61447010247c4936bc9a9fcb2af3d202" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45391 } } peers { permanent_uuid: "14aa376377e34cae9a387069ec429d4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42625 } } }
I20260924 04:08:59.051455 8477 tablet_replica.cc:442] T 6a24fd0824ad438589d227671186deed P 61447010247c4936bc9a9fcb2af3d202: This TxnStatusTablet replica's current role is: LEADER
I20260924 04:08:59.051725 8479 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 04:08:59.051868 8479 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 04:08:59.052563 7925 catalog_manager.cc:5495] T 6a24fd0824ad438589d227671186deed P 61447010247c4936bc9a9fcb2af3d202 reported cstate change: term changed from 0 to 1, leader changed from <none> to 61447010247c4936bc9a9fcb2af3d202 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "61447010247c4936bc9a9fcb2af3d202" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc944248f10b48bc98c503b0f733a0e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "61447010247c4936bc9a9fcb2af3d202" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14aa376377e34cae9a387069ec429d4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42625 } health_report { overall_health: UNKNOWN } } }
I20260924 04:08:59.065500 8436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:59.061570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35938 (local address 127.0.98.2:45391)
0924 04:08:59.061687 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:59.061691 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:59.061703 (+ 12us) server_negotiation.cc:407] Connection header received
0924 04:08:59.061762 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:59.061764 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:59.061812 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:59.061890 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:08:59.062209 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:59.062729 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:59.063952 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:59.064162 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:59.064245 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:59.064560 (+ 315us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:08:59.065060 (+ 500us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:08:59.065202 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:59.065300 (+ 98us) server_negotiation.cc:299] Negotiation successful
0924 04:08:59.065372 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 04:08:59.065512 8471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:59.061278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45391 (local address 127.0.0.1:35938)
0924 04:08:59.061437 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:59.061455 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:59.061593 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:59.061929 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:59.061933 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:59.061944 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 04:08:59.062061 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:59.062070 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:59.062852 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:59.062856 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:59.063813 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:59.063835 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:59.064271 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:59.064275 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:59.064373 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:59.064427 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 04:08:59.065176 (+ 749us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 04:08:59.065193 (+ 17us) client_negotiation.cc:769] Sending connection context
0924 04:08:59.065271 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 04:08:59.065363 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 04:08:59.068508 8083 raft_consensus.cc:1261] T 6a24fd0824ad438589d227671186deed P cc944248f10b48bc98c503b0f733a0e9 [term 1 FOLLOWER]: Refusing update from remote peer 61447010247c4936bc9a9fcb2af3d202: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:08:59.068845 8363 raft_consensus.cc:1261] T 6a24fd0824ad438589d227671186deed P 14aa376377e34cae9a387069ec429d4a [term 1 FOLLOWER]: Refusing update from remote peer 61447010247c4936bc9a9fcb2af3d202: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:08:59.069172 8477 consensus_queue.cc:1035] T 6a24fd0824ad438589d227671186deed P 61447010247c4936bc9a9fcb2af3d202 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc944248f10b48bc98c503b0f733a0e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:08:59.069330 8445 consensus_queue.cc:1035] T 6a24fd0824ad438589d227671186deed P 61447010247c4936bc9a9fcb2af3d202 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14aa376377e34cae9a387069ec429d4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:08:59.071565 8443 tablet_replica.cc:439] T 6a24fd0824ad438589d227671186deed P cc944248f10b48bc98c503b0f733a0e9: TxnStatusTablet state changed. Reason: New leader 61447010247c4936bc9a9fcb2af3d202. Latest consensus state: current_term: 1 leader_uuid: "61447010247c4936bc9a9fcb2af3d202" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc944248f10b48bc98c503b0f733a0e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45551 } } peers { permanent_uuid: "61447010247c4936bc9a9fcb2af3d202" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45391 } } peers { permanent_uuid: "14aa376377e34cae9a387069ec429d4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42625 } } }
I20260924 04:08:59.071688 8443 tablet_replica.cc:442] T 6a24fd0824ad438589d227671186deed P cc944248f10b48bc98c503b0f733a0e9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:08:59.072429 8447 tablet_replica.cc:439] T 6a24fd0824ad438589d227671186deed P 14aa376377e34cae9a387069ec429d4a: TxnStatusTablet state changed. Reason: New leader 61447010247c4936bc9a9fcb2af3d202. Latest consensus state: current_term: 1 leader_uuid: "61447010247c4936bc9a9fcb2af3d202" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc944248f10b48bc98c503b0f733a0e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45551 } } peers { permanent_uuid: "61447010247c4936bc9a9fcb2af3d202" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45391 } } peers { permanent_uuid: "14aa376377e34cae9a387069ec429d4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42625 } } }
I20260924 04:08:59.072556 8447 tablet_replica.cc:442] T 6a24fd0824ad438589d227671186deed P 14aa376377e34cae9a387069ec429d4a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:08:59.073156 8445 tablet_replica.cc:439] T 6a24fd0824ad438589d227671186deed P 61447010247c4936bc9a9fcb2af3d202: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "61447010247c4936bc9a9fcb2af3d202" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc944248f10b48bc98c503b0f733a0e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45551 } } peers { permanent_uuid: "61447010247c4936bc9a9fcb2af3d202" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45391 } } peers { permanent_uuid: "14aa376377e34cae9a387069ec429d4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42625 } } }
I20260924 04:08:59.073282 8445 tablet_replica.cc:442] T 6a24fd0824ad438589d227671186deed P 61447010247c4936bc9a9fcb2af3d202: This TxnStatusTablet replica's current role is: LEADER
I20260924 04:08:59.076254 8447 tablet_replica.cc:439] T 6a24fd0824ad438589d227671186deed P 14aa376377e34cae9a387069ec429d4a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "61447010247c4936bc9a9fcb2af3d202" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc944248f10b48bc98c503b0f733a0e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45551 } } peers { permanent_uuid: "61447010247c4936bc9a9fcb2af3d202" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45391 } } peers { permanent_uuid: "14aa376377e34cae9a387069ec429d4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42625 } } }
I20260924 04:08:59.076361 8447 tablet_replica.cc:442] T 6a24fd0824ad438589d227671186deed P 14aa376377e34cae9a387069ec429d4a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:08:59.077323 8484 tablet_replica.cc:439] T 6a24fd0824ad438589d227671186deed P 61447010247c4936bc9a9fcb2af3d202: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "61447010247c4936bc9a9fcb2af3d202" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc944248f10b48bc98c503b0f733a0e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45551 } } peers { permanent_uuid: "61447010247c4936bc9a9fcb2af3d202" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45391 } } peers { permanent_uuid: "14aa376377e34cae9a387069ec429d4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42625 } } }
I20260924 04:08:59.077545 8484 tablet_replica.cc:442] T 6a24fd0824ad438589d227671186deed P 61447010247c4936bc9a9fcb2af3d202: This TxnStatusTablet replica's current role is: LEADER
I20260924 04:08:59.077685 8451 mvcc.cc:205] Tried to move back new op lower bound from 7332753158419345408 to 7332753158351982592. Current Snapshot: MvccSnapshot[applied={T|T < 7332753158419345408 or (T in {7332753158419345408})}]
I20260924 04:08:59.078526 8443 tablet_replica.cc:439] T 6a24fd0824ad438589d227671186deed P cc944248f10b48bc98c503b0f733a0e9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "61447010247c4936bc9a9fcb2af3d202" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc944248f10b48bc98c503b0f733a0e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45551 } } peers { permanent_uuid: "61447010247c4936bc9a9fcb2af3d202" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45391 } } peers { permanent_uuid: "14aa376377e34cae9a387069ec429d4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42625 } } }
I20260924 04:08:59.078627 8443 tablet_replica.cc:442] T 6a24fd0824ad438589d227671186deed P cc944248f10b48bc98c503b0f733a0e9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:08:59.088454 8415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:59.085051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45391 (local address 127.0.98.3:48699)
0924 04:08:59.085200 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:59.085220 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:59.085289 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:59.085519 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:59.085521 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:59.085526 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 04:08:59.085611 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:59.085615 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:59.086617 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:59.086619 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:59.087241 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:59.087250 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:59.087837 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:59.087840 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:59.087906 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:59.087949 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 04:08:59.088231 (+ 282us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 04:08:59.088235 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 04:08:59.088279 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 04:08:59.088352 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260924 04:08:59.088474 8436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:59.085116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48699 (local address 127.0.98.2:45391)
0924 04:08:59.085229 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:59.085234 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:59.085249 (+ 15us) server_negotiation.cc:407] Connection header received
0924 04:08:59.085375 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:59.085378 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:59.085426 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:59.085504 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:08:59.085697 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:59.086490 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:59.087355 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:59.087726 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:59.087838 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:59.088069 (+ 231us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:08:59.088150 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:08:59.088213 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:59.088291 (+ 78us) server_negotiation.cc:299] Negotiation successful
0924 04:08:59.088343 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 04:08:59.095301 8083 raft_consensus.cc:1261] T c4f1d65d6daa4eeaae734cadcf7d322c P cc944248f10b48bc98c503b0f733a0e9 [term 1 FOLLOWER]: Refusing update from remote peer 14aa376377e34cae9a387069ec429d4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:08:59.095373 8223 raft_consensus.cc:1261] T c4f1d65d6daa4eeaae734cadcf7d322c P 61447010247c4936bc9a9fcb2af3d202 [term 1 FOLLOWER]: Refusing update from remote peer 14aa376377e34cae9a387069ec429d4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:08:59.095675 8447 consensus_queue.cc:1035] T c4f1d65d6daa4eeaae734cadcf7d322c P 14aa376377e34cae9a387069ec429d4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc944248f10b48bc98c503b0f733a0e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:08:59.095964 8442 consensus_queue.cc:1035] T c4f1d65d6daa4eeaae734cadcf7d322c P 14aa376377e34cae9a387069ec429d4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "61447010247c4936bc9a9fcb2af3d202" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:08:59.098624 8450 mvcc.cc:205] Tried to move back new op lower bound from 7332753158531416064 to 7332753157736566784. Current Snapshot: MvccSnapshot[applied={T|T < 7332753158531416064}]
I20260924 04:08:59.116168 8433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:59.112409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50939 (local address 127.0.98.3:42625)
0924 04:08:59.112526 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:59.112530 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:59.112858 (+ 328us) server_negotiation.cc:407] Connection header received
0924 04:08:59.113065 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:59.113069 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:59.113131 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:59.113225 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:08:59.113470 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:59.113980 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:59.114910 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:59.115263 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:59.115374 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:59.115684 (+ 310us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:08:59.115763 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:08:59.115840 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:59.115977 (+ 137us) server_negotiation.cc:299] Negotiation successful
0924 04:08:59.116032 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 04:08:59.116588 8490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:59.112338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42625 (local address 127.0.98.2:50939)
0924 04:08:59.112770 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:59.112785 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:08:59.112931 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:08:59.113223 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:08:59.113226 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:08:59.113234 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 04:08:59.113362 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:59.113369 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:59.114101 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:59.114105 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:59.114790 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:08:59.114802 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:08:59.115374 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:08:59.115380 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:08:59.115486 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:59.115553 (+ 67us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 04:08:59.115879 (+ 326us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 04:08:59.115886 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 04:08:59.116358 (+ 472us) client_negotiation.cc:240] Negotiation successful
0924 04:08:59.116436 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":136,"threads_started":1}
I20260924 04:08:59.137446 7926 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35020:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:08:59.137643 7926 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35020:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:08:59.139652 7926 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 37584f0cfe7049d291a195adfd8d7aaf: Sending DeleteTablet for 3 replicas of tablet c4f1d65d6daa4eeaae734cadcf7d322c
I20260924 04:08:59.140240 8063 tablet_service.cc:1514] Processing DeleteTablet for tablet c4f1d65d6daa4eeaae734cadcf7d322c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:08:59 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38806
I20260924 04:08:59.140328 8203 tablet_service.cc:1514] Processing DeleteTablet for tablet c4f1d65d6daa4eeaae734cadcf7d322c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:08:59 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35930
I20260924 04:08:59.140456 8461 tablet_replica.cc:330] T c4f1d65d6daa4eeaae734cadcf7d322c P cc944248f10b48bc98c503b0f733a0e9: stopping tablet replica
I20260924 04:08:59.140519 8460 tablet_replica.cc:330] T c4f1d65d6daa4eeaae734cadcf7d322c P 61447010247c4936bc9a9fcb2af3d202: stopping tablet replica
I20260924 04:08:59.140573 8461 raft_consensus.cc:2229] T c4f1d65d6daa4eeaae734cadcf7d322c P cc944248f10b48bc98c503b0f733a0e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:08:59.140637 8460 raft_consensus.cc:2229] T c4f1d65d6daa4eeaae734cadcf7d322c P 61447010247c4936bc9a9fcb2af3d202 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:08:59.140666 8461 raft_consensus.cc:2258] T c4f1d65d6daa4eeaae734cadcf7d322c P cc944248f10b48bc98c503b0f733a0e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:08:59.140769 8460 raft_consensus.cc:2258] T c4f1d65d6daa4eeaae734cadcf7d322c P 61447010247c4936bc9a9fcb2af3d202 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:08:59.140880 8343 tablet_service.cc:1514] Processing DeleteTablet for tablet c4f1d65d6daa4eeaae734cadcf7d322c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:08:59 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57816
I20260924 04:08:59.141043 8459 tablet_replica.cc:330] T c4f1d65d6daa4eeaae734cadcf7d322c P 14aa376377e34cae9a387069ec429d4a: stopping tablet replica
I20260924 04:08:59.141144 8459 raft_consensus.cc:2229] T c4f1d65d6daa4eeaae734cadcf7d322c P 14aa376377e34cae9a387069ec429d4a [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:08:59.141332 8459 raft_consensus.cc:2258] T c4f1d65d6daa4eeaae734cadcf7d322c P 14aa376377e34cae9a387069ec429d4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:08:59.141637 8460 ts_tablet_manager.cc:1905] T c4f1d65d6daa4eeaae734cadcf7d322c P 61447010247c4936bc9a9fcb2af3d202: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:08:59.142000 8461 ts_tablet_manager.cc:1905] T c4f1d65d6daa4eeaae734cadcf7d322c P cc944248f10b48bc98c503b0f733a0e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:08:59.142105 8459 ts_tablet_manager.cc:1905] T c4f1d65d6daa4eeaae734cadcf7d322c P 14aa376377e34cae9a387069ec429d4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:08:59.144572 8460 ts_tablet_manager.cc:1918] T c4f1d65d6daa4eeaae734cadcf7d322c P 61447010247c4936bc9a9fcb2af3d202: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 04:08:59.144593 8459 ts_tablet_manager.cc:1918] T c4f1d65d6daa4eeaae734cadcf7d322c P 14aa376377e34cae9a387069ec429d4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 04:08:59.144593 8461 ts_tablet_manager.cc:1918] T c4f1d65d6daa4eeaae734cadcf7d322c P cc944248f10b48bc98c503b0f733a0e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 04:08:59.144639 8460 log.cc:1196] T c4f1d65d6daa4eeaae734cadcf7d322c P 61447010247c4936bc9a9fcb2af3d202: Deleting WAL directory at /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/minicluster-data/ts-1/wal/wals/c4f1d65d6daa4eeaae734cadcf7d322c
I20260924 04:08:59.144661 8459 log.cc:1196] T c4f1d65d6daa4eeaae734cadcf7d322c P 14aa376377e34cae9a387069ec429d4a: Deleting WAL directory at /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/minicluster-data/ts-2/wal/wals/c4f1d65d6daa4eeaae734cadcf7d322c
I20260924 04:08:59.144661 8461 log.cc:1196] T c4f1d65d6daa4eeaae734cadcf7d322c P cc944248f10b48bc98c503b0f733a0e9: Deleting WAL directory at /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/minicluster-data/ts-0/wal/wals/c4f1d65d6daa4eeaae734cadcf7d322c
I20260924 04:08:59.144953 8461 ts_tablet_manager.cc:1939] T c4f1d65d6daa4eeaae734cadcf7d322c P cc944248f10b48bc98c503b0f733a0e9: Deleting consensus metadata
I20260924 04:08:59.144953 8459 ts_tablet_manager.cc:1939] T c4f1d65d6daa4eeaae734cadcf7d322c P 14aa376377e34cae9a387069ec429d4a: Deleting consensus metadata
I20260924 04:08:59.145031 8460 ts_tablet_manager.cc:1939] T c4f1d65d6daa4eeaae734cadcf7d322c P 61447010247c4936bc9a9fcb2af3d202: Deleting consensus metadata
I20260924 04:08:59.145884 7911 catalog_manager.cc:4841] TS cc944248f10b48bc98c503b0f733a0e9 (127.0.98.1:45551): tablet c4f1d65d6daa4eeaae734cadcf7d322c (table test-table [id=b207630f9223449d8456bf2fddd51310]) successfully deleted
I20260924 04:08:59.146071 7911 catalog_manager.cc:4841] TS 61447010247c4936bc9a9fcb2af3d202 (127.0.98.2:45391): tablet c4f1d65d6daa4eeaae734cadcf7d322c (table test-table [id=b207630f9223449d8456bf2fddd51310]) successfully deleted
I20260924 04:08:59.146469 7910 catalog_manager.cc:4841] TS 14aa376377e34cae9a387069ec429d4a (127.0.98.3:42625): tablet c4f1d65d6daa4eeaae734cadcf7d322c (table test-table [id=b207630f9223449d8456bf2fddd51310]) successfully deleted
Sep 24 04:08:59 dist-test-slave-00ld krb5kdc[7881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222939, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 04:08:59 dist-test-slave-00ld krb5kdc[7881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:08:59 dist-test-slave-00ld krb5kdc[7881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 04:08:59.171034 7975 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:08:59.164052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35042 (local address 127.0.98.62:32951)
0924 04:08:59.164194 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:59.164200 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:59.164345 (+ 145us) server_negotiation.cc:407] Connection header received
0924 04:08:59.164453 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:59.164456 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:59.164509 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:59.164606 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:59.165568 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:59.166435 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:59.167368 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:59.167716 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:59.167811 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:59.170905 (+ 3094us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35042: BlockingRecv error: recv got EOF from 127.0.0.1:35042 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 24 04:08:59 dist-test-slave-00ld krb5kdc[7881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222939, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 04:08:59.181416 7975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:59.172436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35050 (local address 127.0.98.62:32951)
0924 04:08:59.172550 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:59.172555 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:59.172900 (+ 345us) server_negotiation.cc:407] Connection header received
0924 04:08:59.173028 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:59.173030 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:59.173071 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:59.173133 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:08:59.173814 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:59.174316 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:59.175398 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:59.175614 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:59.175691 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:59.178266 (+ 2575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:08:59.178269 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:08:59.178271 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:08:59.178295 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:08:59.179702 (+ 1407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:59.180107 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:59.180115 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:59.180116 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:59.180187 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:08:59.180437 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:08:59.180442 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:08:59.180444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:08:59.180624 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:08:59.180796 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:59.181029 (+ 233us) server_negotiation.cc:299] Negotiation successful
0924 04:08:59.181143 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 04:08:59.184116 7926 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35050:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:08:59.184551 7926 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:08:59.189234 8063 tablet_service.cc:1467] Processing CreateTablet for tablet 01054af416174baf87c42c3c85344f48 (DEFAULT_TABLE table=test-table [id=977c5317c96a4ecf889e6928c1551606]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:08:59.189301 8203 tablet_service.cc:1467] Processing CreateTablet for tablet 01054af416174baf87c42c3c85344f48 (DEFAULT_TABLE table=test-table [id=977c5317c96a4ecf889e6928c1551606]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:08:59.189575 8203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01054af416174baf87c42c3c85344f48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:08:59.189517 8343 tablet_service.cc:1467] Processing CreateTablet for tablet 01054af416174baf87c42c3c85344f48 (DEFAULT_TABLE table=test-table [id=977c5317c96a4ecf889e6928c1551606]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:08:59.189568 8063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01054af416174baf87c42c3c85344f48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:08:59.189760 8343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01054af416174baf87c42c3c85344f48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:08:59.192112 8441 tablet_bootstrap.cc:492] T 01054af416174baf87c42c3c85344f48 P 61447010247c4936bc9a9fcb2af3d202: Bootstrap starting.
I20260924 04:08:59.192421 8437 tablet_bootstrap.cc:492] T 01054af416174baf87c42c3c85344f48 P cc944248f10b48bc98c503b0f733a0e9: Bootstrap starting.
I20260924 04:08:59.193117 8441 tablet_bootstrap.cc:654] T 01054af416174baf87c42c3c85344f48 P 61447010247c4936bc9a9fcb2af3d202: Neither blocks nor log segments found. Creating new log.
I20260924 04:08:59.193317 8437 tablet_bootstrap.cc:654] T 01054af416174baf87c42c3c85344f48 P cc944248f10b48bc98c503b0f733a0e9: Neither blocks nor log segments found. Creating new log.
I20260924 04:08:59.193853 8438 tablet_bootstrap.cc:492] T 01054af416174baf87c42c3c85344f48 P 14aa376377e34cae9a387069ec429d4a: Bootstrap starting.
I20260924 04:08:59.194523 8441 tablet_bootstrap.cc:492] T 01054af416174baf87c42c3c85344f48 P 61447010247c4936bc9a9fcb2af3d202: No bootstrap required, opened a new log
I20260924 04:08:59.194605 8441 ts_tablet_manager.cc:1397] T 01054af416174baf87c42c3c85344f48 P 61447010247c4936bc9a9fcb2af3d202: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260924 04:08:59.194728 8438 tablet_bootstrap.cc:654] T 01054af416174baf87c42c3c85344f48 P 14aa376377e34cae9a387069ec429d4a: Neither blocks nor log segments found. Creating new log.
I20260924 04:08:59.195469 8441 raft_consensus.cc:348] T 01054af416174baf87c42c3c85344f48 P 61447010247c4936bc9a9fcb2af3d202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc944248f10b48bc98c503b0f733a0e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45551 } } peers { permanent_uuid: "61447010247c4936bc9a9fcb2af3d202" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45391 } } peers { permanent_uuid: "14aa376377e34cae9a387069ec429d4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42625 } }
I20260924 04:08:59.195611 8441 raft_consensus.cc:374] T 01054af416174baf87c42c3c85344f48 P 61447010247c4936bc9a9fcb2af3d202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:08:59.195657 8441 raft_consensus.cc:729] T 01054af416174baf87c42c3c85344f48 P 61447010247c4936bc9a9fcb2af3d202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61447010247c4936bc9a9fcb2af3d202, State: Initialized, Role: FOLLOWER
I20260924 04:08:59.195873 8437 tablet_bootstrap.cc:492] T 01054af416174baf87c42c3c85344f48 P cc944248f10b48bc98c503b0f733a0e9: No bootstrap required, opened a new log
I20260924 04:08:59.195854 8441 consensus_queue.cc:260] T 01054af416174baf87c42c3c85344f48 P 61447010247c4936bc9a9fcb2af3d202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc944248f10b48bc98c503b0f733a0e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45551 } } peers { permanent_uuid: "61447010247c4936bc9a9fcb2af3d202" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45391 } } peers { permanent_uuid: "14aa376377e34cae9a387069ec429d4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42625 } }
I20260924 04:08:59.195953 8437 ts_tablet_manager.cc:1397] T 01054af416174baf87c42c3c85344f48 P cc944248f10b48bc98c503b0f733a0e9: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260924 04:08:59.196378 8437 raft_consensus.cc:348] T 01054af416174baf87c42c3c85344f48 P cc944248f10b48bc98c503b0f733a0e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc944248f10b48bc98c503b0f733a0e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45551 } } peers { permanent_uuid: "61447010247c4936bc9a9fcb2af3d202" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45391 } } peers { permanent_uuid: "14aa376377e34cae9a387069ec429d4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42625 } }
I20260924 04:08:59.196489 8437 raft_consensus.cc:374] T 01054af416174baf87c42c3c85344f48 P cc944248f10b48bc98c503b0f733a0e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:08:59.196522 8437 raft_consensus.cc:729] T 01054af416174baf87c42c3c85344f48 P cc944248f10b48bc98c503b0f733a0e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc944248f10b48bc98c503b0f733a0e9, State: Initialized, Role: FOLLOWER
I20260924 04:08:59.196587 8441 ts_tablet_manager.cc:1428] T 01054af416174baf87c42c3c85344f48 P 61447010247c4936bc9a9fcb2af3d202: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260924 04:08:59.196636 8437 consensus_queue.cc:260] T 01054af416174baf87c42c3c85344f48 P cc944248f10b48bc98c503b0f733a0e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc944248f10b48bc98c503b0f733a0e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45551 } } peers { permanent_uuid: "61447010247c4936bc9a9fcb2af3d202" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45391 } } peers { permanent_uuid: "14aa376377e34cae9a387069ec429d4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42625 } }
I20260924 04:08:59.196913 8437 ts_tablet_manager.cc:1428] T 01054af416174baf87c42c3c85344f48 P cc944248f10b48bc98c503b0f733a0e9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:08:59.197664 8438 tablet_bootstrap.cc:492] T 01054af416174baf87c42c3c85344f48 P 14aa376377e34cae9a387069ec429d4a: No bootstrap required, opened a new log
I20260924 04:08:59.197772 8438 ts_tablet_manager.cc:1397] T 01054af416174baf87c42c3c85344f48 P 14aa376377e34cae9a387069ec429d4a: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260924 04:08:59.198220 8438 raft_consensus.cc:348] T 01054af416174baf87c42c3c85344f48 P 14aa376377e34cae9a387069ec429d4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc944248f10b48bc98c503b0f733a0e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45551 } } peers { permanent_uuid: "61447010247c4936bc9a9fcb2af3d202" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45391 } } peers { permanent_uuid: "14aa376377e34cae9a387069ec429d4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42625 } }
I20260924 04:08:59.198334 8438 raft_consensus.cc:374] T 01054af416174baf87c42c3c85344f48 P 14aa376377e34cae9a387069ec429d4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:08:59.198371 8438 raft_consensus.cc:729] T 01054af416174baf87c42c3c85344f48 P 14aa376377e34cae9a387069ec429d4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14aa376377e34cae9a387069ec429d4a, State: Initialized, Role: FOLLOWER
I20260924 04:08:59.198482 8438 consensus_queue.cc:260] T 01054af416174baf87c42c3c85344f48 P 14aa376377e34cae9a387069ec429d4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc944248f10b48bc98c503b0f733a0e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45551 } } peers { permanent_uuid: "61447010247c4936bc9a9fcb2af3d202" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45391 } } peers { permanent_uuid: "14aa376377e34cae9a387069ec429d4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42625 } }
I20260924 04:08:59.198747 8438 ts_tablet_manager.cc:1428] T 01054af416174baf87c42c3c85344f48 P 14aa376377e34cae9a387069ec429d4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:08:59.248175 8484 raft_consensus.cc:482] T 01054af416174baf87c42c3c85344f48 P 61447010247c4936bc9a9fcb2af3d202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:08:59.248343 8484 raft_consensus.cc:504] T 01054af416174baf87c42c3c85344f48 P 61447010247c4936bc9a9fcb2af3d202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc944248f10b48bc98c503b0f733a0e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45551 } } peers { permanent_uuid: "61447010247c4936bc9a9fcb2af3d202" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45391 } } peers { permanent_uuid: "14aa376377e34cae9a387069ec429d4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42625 } }
I20260924 04:08:59.248791 8484 leader_election.cc:288] T 01054af416174baf87c42c3c85344f48 P 61447010247c4936bc9a9fcb2af3d202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc944248f10b48bc98c503b0f733a0e9 (127.0.98.1:45551), 14aa376377e34cae9a387069ec429d4a (127.0.98.3:42625)
I20260924 04:08:59.249207 8083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01054af416174baf87c42c3c85344f48" candidate_uuid: "61447010247c4936bc9a9fcb2af3d202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc944248f10b48bc98c503b0f733a0e9" is_pre_election: true
I20260924 04:08:59.249208 8363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01054af416174baf87c42c3c85344f48" candidate_uuid: "61447010247c4936bc9a9fcb2af3d202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14aa376377e34cae9a387069ec429d4a" is_pre_election: true
I20260924 04:08:59.249341 8083 raft_consensus.cc:2454] T 01054af416174baf87c42c3c85344f48 P cc944248f10b48bc98c503b0f733a0e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61447010247c4936bc9a9fcb2af3d202 in term 0.
I20260924 04:08:59.249366 8363 raft_consensus.cc:2454] T 01054af416174baf87c42c3c85344f48 P 14aa376377e34cae9a387069ec429d4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61447010247c4936bc9a9fcb2af3d202 in term 0.
I20260924 04:08:59.249687 8154 leader_election.cc:302] T 01054af416174baf87c42c3c85344f48 P 61447010247c4936bc9a9fcb2af3d202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14aa376377e34cae9a387069ec429d4a, 61447010247c4936bc9a9fcb2af3d202; no voters:
I20260924 04:08:59.249858 8484 raft_consensus.cc:2781] T 01054af416174baf87c42c3c85344f48 P 61447010247c4936bc9a9fcb2af3d202 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:08:59.249930 8484 raft_consensus.cc:482] T 01054af416174baf87c42c3c85344f48 P 61447010247c4936bc9a9fcb2af3d202 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:08:59.249984 8484 raft_consensus.cc:3037] T 01054af416174baf87c42c3c85344f48 P 61447010247c4936bc9a9fcb2af3d202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:08:59.251070 8484 raft_consensus.cc:504] T 01054af416174baf87c42c3c85344f48 P 61447010247c4936bc9a9fcb2af3d202 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc944248f10b48bc98c503b0f733a0e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45551 } } peers { permanent_uuid: "61447010247c4936bc9a9fcb2af3d202" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45391 } } peers { permanent_uuid: "14aa376377e34cae9a387069ec429d4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42625 } }
I20260924 04:08:59.251436 8484 leader_election.cc:288] T 01054af416174baf87c42c3c85344f48 P 61447010247c4936bc9a9fcb2af3d202 [CANDIDATE]: Term 1 election: Requested vote from peers cc944248f10b48bc98c503b0f733a0e9 (127.0.98.1:45551), 14aa376377e34cae9a387069ec429d4a (127.0.98.3:42625)
I20260924 04:08:59.251786 8083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01054af416174baf87c42c3c85344f48" candidate_uuid: "61447010247c4936bc9a9fcb2af3d202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc944248f10b48bc98c503b0f733a0e9"
I20260924 04:08:59.251791 8363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01054af416174baf87c42c3c85344f48" candidate_uuid: "61447010247c4936bc9a9fcb2af3d202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14aa376377e34cae9a387069ec429d4a"
I20260924 04:08:59.251909 8083 raft_consensus.cc:3037] T 01054af416174baf87c42c3c85344f48 P cc944248f10b48bc98c503b0f733a0e9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:08:59.251919 8363 raft_consensus.cc:3037] T 01054af416174baf87c42c3c85344f48 P 14aa376377e34cae9a387069ec429d4a [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:08:59.253062 8363 raft_consensus.cc:2454] T 01054af416174baf87c42c3c85344f48 P 14aa376377e34cae9a387069ec429d4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61447010247c4936bc9a9fcb2af3d202 in term 1.
I20260924 04:08:59.253062 8083 raft_consensus.cc:2454] T 01054af416174baf87c42c3c85344f48 P cc944248f10b48bc98c503b0f733a0e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61447010247c4936bc9a9fcb2af3d202 in term 1.
I20260924 04:08:59.253369 8154 leader_election.cc:302] T 01054af416174baf87c42c3c85344f48 P 61447010247c4936bc9a9fcb2af3d202 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14aa376377e34cae9a387069ec429d4a, 61447010247c4936bc9a9fcb2af3d202; no voters:
I20260924 04:08:59.253541 8484 raft_consensus.cc:2781] T 01054af416174baf87c42c3c85344f48 P 61447010247c4936bc9a9fcb2af3d202 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:08:59.253644 8484 raft_consensus.cc:686] T 01054af416174baf87c42c3c85344f48 P 61447010247c4936bc9a9fcb2af3d202 [term 1 LEADER]: Becoming Leader. State: Replica: 61447010247c4936bc9a9fcb2af3d202, State: Running, Role: LEADER
I20260924 04:08:59.253782 8484 consensus_queue.cc:237] T 01054af416174baf87c42c3c85344f48 P 61447010247c4936bc9a9fcb2af3d202 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc944248f10b48bc98c503b0f733a0e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45551 } } peers { permanent_uuid: "61447010247c4936bc9a9fcb2af3d202" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45391 } } peers { permanent_uuid: "14aa376377e34cae9a387069ec429d4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42625 } }
I20260924 04:08:59.255323 7925 catalog_manager.cc:5495] T 01054af416174baf87c42c3c85344f48 P 61447010247c4936bc9a9fcb2af3d202 reported cstate change: term changed from 0 to 1, leader changed from <none> to 61447010247c4936bc9a9fcb2af3d202 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "61447010247c4936bc9a9fcb2af3d202" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc944248f10b48bc98c503b0f733a0e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "61447010247c4936bc9a9fcb2af3d202" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14aa376377e34cae9a387069ec429d4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42625 } health_report { overall_health: UNKNOWN } } }
I20260924 04:08:59.279484 8436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:59.276532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35944 (local address 127.0.98.2:45391)
0924 04:08:59.276719 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:59.276724 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:59.276739 (+ 15us) server_negotiation.cc:407] Connection header received
0924 04:08:59.276823 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:59.276825 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:59.276866 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:59.276928 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:08:59.277152 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:59.277649 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:59.278520 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:59.278726 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:59.278798 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:59.279024 (+ 226us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:08:59.279093 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:08:59.279147 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:59.279251 (+ 104us) server_negotiation.cc:299] Negotiation successful
0924 04:08:59.279292 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 04:08:59.281769 8363 raft_consensus.cc:1261] T 01054af416174baf87c42c3c85344f48 P 14aa376377e34cae9a387069ec429d4a [term 1 FOLLOWER]: Refusing update from remote peer 61447010247c4936bc9a9fcb2af3d202: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:08:59.281824 8083 raft_consensus.cc:1261] T 01054af416174baf87c42c3c85344f48 P cc944248f10b48bc98c503b0f733a0e9 [term 1 FOLLOWER]: Refusing update from remote peer 61447010247c4936bc9a9fcb2af3d202: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:08:59.282264 8484 consensus_queue.cc:1035] T 01054af416174baf87c42c3c85344f48 P 61447010247c4936bc9a9fcb2af3d202 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc944248f10b48bc98c503b0f733a0e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:08:59.282430 8477 consensus_queue.cc:1035] T 01054af416174baf87c42c3c85344f48 P 61447010247c4936bc9a9fcb2af3d202 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14aa376377e34cae9a387069ec429d4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:08:59.291676 7924 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:08:59.291828 7924 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:08:59.293864 7924 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 37584f0cfe7049d291a195adfd8d7aaf: Sending DeleteTablet for 3 replicas of tablet 01054af416174baf87c42c3c85344f48
I20260924 04:08:59.294447 8203 tablet_service.cc:1514] Processing DeleteTablet for tablet 01054af416174baf87c42c3c85344f48 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:08:59 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35930
I20260924 04:08:59.294485 8063 tablet_service.cc:1514] Processing DeleteTablet for tablet 01054af416174baf87c42c3c85344f48 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:08:59 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38806
I20260924 04:08:59.294636 8460 tablet_replica.cc:330] T 01054af416174baf87c42c3c85344f48 P 61447010247c4936bc9a9fcb2af3d202: stopping tablet replica
I20260924 04:08:59.294677 8461 tablet_replica.cc:330] T 01054af416174baf87c42c3c85344f48 P cc944248f10b48bc98c503b0f733a0e9: stopping tablet replica
I20260924 04:08:59.294759 8460 raft_consensus.cc:2229] T 01054af416174baf87c42c3c85344f48 P 61447010247c4936bc9a9fcb2af3d202 [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:08:59.294791 8461 raft_consensus.cc:2229] T 01054af416174baf87c42c3c85344f48 P cc944248f10b48bc98c503b0f733a0e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:08:59.294924 8461 raft_consensus.cc:2258] T 01054af416174baf87c42c3c85344f48 P cc944248f10b48bc98c503b0f733a0e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:08:59.294960 8343 tablet_service.cc:1514] Processing DeleteTablet for tablet 01054af416174baf87c42c3c85344f48 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:08:59 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57816
I20260924 04:08:59.295094 8460 raft_consensus.cc:2258] T 01054af416174baf87c42c3c85344f48 P 61447010247c4936bc9a9fcb2af3d202 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:08:59.295101 7924 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35050:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 04:08:59.295215 8459 tablet_replica.cc:330] T 01054af416174baf87c42c3c85344f48 P 14aa376377e34cae9a387069ec429d4a: stopping tablet replica
I20260924 04:08:59.295341 8459 raft_consensus.cc:2229] T 01054af416174baf87c42c3c85344f48 P 14aa376377e34cae9a387069ec429d4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:08:59.295449 8459 raft_consensus.cc:2258] T 01054af416174baf87c42c3c85344f48 P 14aa376377e34cae9a387069ec429d4a [term 1 FOLLOWER]: Raft consensus is shut down!
W20260924 04:08:59.295459 7924 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:08:59.296298 8459 ts_tablet_manager.cc:1905] T 01054af416174baf87c42c3c85344f48 P 14aa376377e34cae9a387069ec429d4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:08:59.298842 8459 ts_tablet_manager.cc:1918] T 01054af416174baf87c42c3c85344f48 P 14aa376377e34cae9a387069ec429d4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:08:59.298923 8459 log.cc:1196] T 01054af416174baf87c42c3c85344f48 P 14aa376377e34cae9a387069ec429d4a: Deleting WAL directory at /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/minicluster-data/ts-2/wal/wals/01054af416174baf87c42c3c85344f48
I20260924 04:08:59.299191 8459 ts_tablet_manager.cc:1939] T 01054af416174baf87c42c3c85344f48 P 14aa376377e34cae9a387069ec429d4a: Deleting consensus metadata
I20260924 04:08:59.300076 8343 tablet_service.cc:1467] Processing CreateTablet for tablet c54ecfa8868f4e7584a08b83661f2cec (DEFAULT_TABLE table=test-table [id=c1f6791e039548ecafc01e94860b32b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:08:59.300123 8063 tablet_service.cc:1467] Processing CreateTablet for tablet c54ecfa8868f4e7584a08b83661f2cec (DEFAULT_TABLE table=test-table [id=c1f6791e039548ecafc01e94860b32b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:08:59.300310 8343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c54ecfa8868f4e7584a08b83661f2cec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:08:59.300427 8063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c54ecfa8868f4e7584a08b83661f2cec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:08:59.301941 8438 tablet_bootstrap.cc:492] T c54ecfa8868f4e7584a08b83661f2cec P 14aa376377e34cae9a387069ec429d4a: Bootstrap starting.
I20260924 04:08:59.301995 7910 catalog_manager.cc:4841] TS 14aa376377e34cae9a387069ec429d4a (127.0.98.3:42625): tablet 01054af416174baf87c42c3c85344f48 (table test-table [id=977c5317c96a4ecf889e6928c1551606]) successfully deleted
I20260924 04:08:59.302445 8203 tablet_service.cc:1467] Processing CreateTablet for tablet c54ecfa8868f4e7584a08b83661f2cec (DEFAULT_TABLE table=test-table [id=c1f6791e039548ecafc01e94860b32b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:08:59.302697 8203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c54ecfa8868f4e7584a08b83661f2cec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:08:59.302802 8437 tablet_bootstrap.cc:492] T c54ecfa8868f4e7584a08b83661f2cec P cc944248f10b48bc98c503b0f733a0e9: Bootstrap starting.
I20260924 04:08:59.302819 8438 tablet_bootstrap.cc:654] T c54ecfa8868f4e7584a08b83661f2cec P 14aa376377e34cae9a387069ec429d4a: Neither blocks nor log segments found. Creating new log.
I20260924 04:08:59.303627 8437 tablet_bootstrap.cc:654] T c54ecfa8868f4e7584a08b83661f2cec P cc944248f10b48bc98c503b0f733a0e9: Neither blocks nor log segments found. Creating new log.
I20260924 04:08:59.304631 8461 ts_tablet_manager.cc:1905] T 01054af416174baf87c42c3c85344f48 P cc944248f10b48bc98c503b0f733a0e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:08:59.305527 8460 ts_tablet_manager.cc:1905] T 01054af416174baf87c42c3c85344f48 P 61447010247c4936bc9a9fcb2af3d202: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:08:59.306689 8461 ts_tablet_manager.cc:1918] T 01054af416174baf87c42c3c85344f48 P cc944248f10b48bc98c503b0f733a0e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:08:59.306762 8461 log.cc:1196] T 01054af416174baf87c42c3c85344f48 P cc944248f10b48bc98c503b0f733a0e9: Deleting WAL directory at /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/minicluster-data/ts-0/wal/wals/01054af416174baf87c42c3c85344f48
I20260924 04:08:59.307029 8461 ts_tablet_manager.cc:1939] T 01054af416174baf87c42c3c85344f48 P cc944248f10b48bc98c503b0f733a0e9: Deleting consensus metadata
I20260924 04:08:59.307353 8441 tablet_bootstrap.cc:492] T c54ecfa8868f4e7584a08b83661f2cec P 61447010247c4936bc9a9fcb2af3d202: Bootstrap starting.
I20260924 04:08:59.307696 8460 ts_tablet_manager.cc:1918] T 01054af416174baf87c42c3c85344f48 P 61447010247c4936bc9a9fcb2af3d202: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:08:59.307781 8460 log.cc:1196] T 01054af416174baf87c42c3c85344f48 P 61447010247c4936bc9a9fcb2af3d202: Deleting WAL directory at /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/minicluster-data/ts-1/wal/wals/01054af416174baf87c42c3c85344f48
I20260924 04:08:59.308012 8460 ts_tablet_manager.cc:1939] T 01054af416174baf87c42c3c85344f48 P 61447010247c4936bc9a9fcb2af3d202: Deleting consensus metadata
I20260924 04:08:59.308143 8441 tablet_bootstrap.cc:654] T c54ecfa8868f4e7584a08b83661f2cec P 61447010247c4936bc9a9fcb2af3d202: Neither blocks nor log segments found. Creating new log.
I20260924 04:08:59.308650 7911 catalog_manager.cc:4841] TS 61447010247c4936bc9a9fcb2af3d202 (127.0.98.2:45391): tablet 01054af416174baf87c42c3c85344f48 (table test-table [id=977c5317c96a4ecf889e6928c1551606]) successfully deleted
I20260924 04:08:59.308929 8437 tablet_bootstrap.cc:492] T c54ecfa8868f4e7584a08b83661f2cec P cc944248f10b48bc98c503b0f733a0e9: No bootstrap required, opened a new log
I20260924 04:08:59.309015 8437 ts_tablet_manager.cc:1397] T c54ecfa8868f4e7584a08b83661f2cec P cc944248f10b48bc98c503b0f733a0e9: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260924 04:08:59.309293 8438 tablet_bootstrap.cc:492] T c54ecfa8868f4e7584a08b83661f2cec P 14aa376377e34cae9a387069ec429d4a: No bootstrap required, opened a new log
I20260924 04:08:59.309399 8438 ts_tablet_manager.cc:1397] T c54ecfa8868f4e7584a08b83661f2cec P 14aa376377e34cae9a387069ec429d4a: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260924 04:08:59.309420 8437 raft_consensus.cc:348] T c54ecfa8868f4e7584a08b83661f2cec P cc944248f10b48bc98c503b0f733a0e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc944248f10b48bc98c503b0f733a0e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45551 } } peers { permanent_uuid: "61447010247c4936bc9a9fcb2af3d202" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45391 } } peers { permanent_uuid: "14aa376377e34cae9a387069ec429d4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42625 } }
I20260924 04:08:59.309538 8437 raft_consensus.cc:374] T c54ecfa8868f4e7584a08b83661f2cec P cc944248f10b48bc98c503b0f733a0e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:08:59.309579 8437 raft_consensus.cc:729] T c54ecfa8868f4e7584a08b83661f2cec P cc944248f10b48bc98c503b0f733a0e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc944248f10b48bc98c503b0f733a0e9, State: Initialized, Role: FOLLOWER
I20260924 04:08:59.309695 8437 consensus_queue.cc:260] T c54ecfa8868f4e7584a08b83661f2cec P cc944248f10b48bc98c503b0f733a0e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc944248f10b48bc98c503b0f733a0e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45551 } } peers { permanent_uuid: "61447010247c4936bc9a9fcb2af3d202" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45391 } } peers { permanent_uuid: "14aa376377e34cae9a387069ec429d4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42625 } }
I20260924 04:08:59.309832 8438 raft_consensus.cc:348] T c54ecfa8868f4e7584a08b83661f2cec P 14aa376377e34cae9a387069ec429d4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc944248f10b48bc98c503b0f733a0e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45551 } } peers { permanent_uuid: "61447010247c4936bc9a9fcb2af3d202" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45391 } } peers { permanent_uuid: "14aa376377e34cae9a387069ec429d4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42625 } }
I20260924 04:08:59.309944 8438 raft_consensus.cc:374] T c54ecfa8868f4e7584a08b83661f2cec P 14aa376377e34cae9a387069ec429d4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:08:59.309978 8437 ts_tablet_manager.cc:1428] T c54ecfa8868f4e7584a08b83661f2cec P cc944248f10b48bc98c503b0f733a0e9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 04:08:59.309986 8438 raft_consensus.cc:729] T c54ecfa8868f4e7584a08b83661f2cec P 14aa376377e34cae9a387069ec429d4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14aa376377e34cae9a387069ec429d4a, State: Initialized, Role: FOLLOWER
I20260924 04:08:59.310104 8438 consensus_queue.cc:260] T c54ecfa8868f4e7584a08b83661f2cec P 14aa376377e34cae9a387069ec429d4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc944248f10b48bc98c503b0f733a0e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45551 } } peers { permanent_uuid: "61447010247c4936bc9a9fcb2af3d202" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45391 } } peers { permanent_uuid: "14aa376377e34cae9a387069ec429d4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42625 } }
I20260924 04:08:59.310338 8438 ts_tablet_manager.cc:1428] T c54ecfa8868f4e7584a08b83661f2cec P 14aa376377e34cae9a387069ec429d4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:08:59.310761 7911 catalog_manager.cc:4841] TS cc944248f10b48bc98c503b0f733a0e9 (127.0.98.1:45551): tablet 01054af416174baf87c42c3c85344f48 (table test-table [id=977c5317c96a4ecf889e6928c1551606]) successfully deleted
I20260924 04:08:59.311579 8441 tablet_bootstrap.cc:492] T c54ecfa8868f4e7584a08b83661f2cec P 61447010247c4936bc9a9fcb2af3d202: No bootstrap required, opened a new log
I20260924 04:08:59.311663 8441 ts_tablet_manager.cc:1397] T c54ecfa8868f4e7584a08b83661f2cec P 61447010247c4936bc9a9fcb2af3d202: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260924 04:08:59.312119 8441 raft_consensus.cc:348] T c54ecfa8868f4e7584a08b83661f2cec P 61447010247c4936bc9a9fcb2af3d202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc944248f10b48bc98c503b0f733a0e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45551 } } peers { permanent_uuid: "61447010247c4936bc9a9fcb2af3d202" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45391 } } peers { permanent_uuid: "14aa376377e34cae9a387069ec429d4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42625 } }
I20260924 04:08:59.312228 8441 raft_consensus.cc:374] T c54ecfa8868f4e7584a08b83661f2cec P 61447010247c4936bc9a9fcb2af3d202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:08:59.312270 8441 raft_consensus.cc:729] T c54ecfa8868f4e7584a08b83661f2cec P 61447010247c4936bc9a9fcb2af3d202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61447010247c4936bc9a9fcb2af3d202, State: Initialized, Role: FOLLOWER
I20260924 04:08:59.312407 8441 consensus_queue.cc:260] T c54ecfa8868f4e7584a08b83661f2cec P 61447010247c4936bc9a9fcb2af3d202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc944248f10b48bc98c503b0f733a0e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45551 } } peers { permanent_uuid: "61447010247c4936bc9a9fcb2af3d202" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45391 } } peers { permanent_uuid: "14aa376377e34cae9a387069ec429d4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42625 } }
I20260924 04:08:59.312711 8441 ts_tablet_manager.cc:1428] T c54ecfa8868f4e7584a08b83661f2cec P 61447010247c4936bc9a9fcb2af3d202: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:08:59.427071 8442 raft_consensus.cc:482] T c54ecfa8868f4e7584a08b83661f2cec P 14aa376377e34cae9a387069ec429d4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:08:59.427222 8442 raft_consensus.cc:504] T c54ecfa8868f4e7584a08b83661f2cec P 14aa376377e34cae9a387069ec429d4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc944248f10b48bc98c503b0f733a0e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45551 } } peers { permanent_uuid: "61447010247c4936bc9a9fcb2af3d202" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45391 } } peers { permanent_uuid: "14aa376377e34cae9a387069ec429d4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42625 } }
I20260924 04:08:59.427659 8442 leader_election.cc:288] T c54ecfa8868f4e7584a08b83661f2cec P 14aa376377e34cae9a387069ec429d4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc944248f10b48bc98c503b0f733a0e9 (127.0.98.1:45551), 61447010247c4936bc9a9fcb2af3d202 (127.0.98.2:45391)
I20260924 04:08:59.428092 8083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c54ecfa8868f4e7584a08b83661f2cec" candidate_uuid: "14aa376377e34cae9a387069ec429d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc944248f10b48bc98c503b0f733a0e9" is_pre_election: true
I20260924 04:08:59.428195 8223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c54ecfa8868f4e7584a08b83661f2cec" candidate_uuid: "14aa376377e34cae9a387069ec429d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61447010247c4936bc9a9fcb2af3d202" is_pre_election: true
I20260924 04:08:59.428268 8083 raft_consensus.cc:2454] T c54ecfa8868f4e7584a08b83661f2cec P cc944248f10b48bc98c503b0f733a0e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14aa376377e34cae9a387069ec429d4a in term 0.
I20260924 04:08:59.428349 8223 raft_consensus.cc:2454] T c54ecfa8868f4e7584a08b83661f2cec P 61447010247c4936bc9a9fcb2af3d202 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14aa376377e34cae9a387069ec429d4a in term 0.
I20260924 04:08:59.428622 8295 leader_election.cc:302] T c54ecfa8868f4e7584a08b83661f2cec P 14aa376377e34cae9a387069ec429d4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14aa376377e34cae9a387069ec429d4a, cc944248f10b48bc98c503b0f733a0e9; no voters:
I20260924 04:08:59.428882 8442 raft_consensus.cc:2781] T c54ecfa8868f4e7584a08b83661f2cec P 14aa376377e34cae9a387069ec429d4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:08:59.428961 8442 raft_consensus.cc:482] T c54ecfa8868f4e7584a08b83661f2cec P 14aa376377e34cae9a387069ec429d4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:08:59.429018 8442 raft_consensus.cc:3037] T c54ecfa8868f4e7584a08b83661f2cec P 14aa376377e34cae9a387069ec429d4a [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:08:59.430027 8442 raft_consensus.cc:504] T c54ecfa8868f4e7584a08b83661f2cec P 14aa376377e34cae9a387069ec429d4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc944248f10b48bc98c503b0f733a0e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45551 } } peers { permanent_uuid: "61447010247c4936bc9a9fcb2af3d202" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45391 } } peers { permanent_uuid: "14aa376377e34cae9a387069ec429d4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42625 } }
I20260924 04:08:59.430364 8442 leader_election.cc:288] T c54ecfa8868f4e7584a08b83661f2cec P 14aa376377e34cae9a387069ec429d4a [CANDIDATE]: Term 1 election: Requested vote from peers cc944248f10b48bc98c503b0f733a0e9 (127.0.98.1:45551), 61447010247c4936bc9a9fcb2af3d202 (127.0.98.2:45391)
I20260924 04:08:59.430764 8223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c54ecfa8868f4e7584a08b83661f2cec" candidate_uuid: "14aa376377e34cae9a387069ec429d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61447010247c4936bc9a9fcb2af3d202"
I20260924 04:08:59.430789 8083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c54ecfa8868f4e7584a08b83661f2cec" candidate_uuid: "14aa376377e34cae9a387069ec429d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc944248f10b48bc98c503b0f733a0e9"
I20260924 04:08:59.430894 8223 raft_consensus.cc:3037] T c54ecfa8868f4e7584a08b83661f2cec P 61447010247c4936bc9a9fcb2af3d202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:08:59.430918 8083 raft_consensus.cc:3037] T c54ecfa8868f4e7584a08b83661f2cec P cc944248f10b48bc98c503b0f733a0e9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:08:59.431813 8083 raft_consensus.cc:2454] T c54ecfa8868f4e7584a08b83661f2cec P cc944248f10b48bc98c503b0f733a0e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14aa376377e34cae9a387069ec429d4a in term 1.
I20260924 04:08:59.432019 8223 raft_consensus.cc:2454] T c54ecfa8868f4e7584a08b83661f2cec P 61447010247c4936bc9a9fcb2af3d202 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14aa376377e34cae9a387069ec429d4a in term 1.
I20260924 04:08:59.432089 8295 leader_election.cc:302] T c54ecfa8868f4e7584a08b83661f2cec P 14aa376377e34cae9a387069ec429d4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14aa376377e34cae9a387069ec429d4a, cc944248f10b48bc98c503b0f733a0e9; no voters:
I20260924 04:08:59.432255 8442 raft_consensus.cc:2781] T c54ecfa8868f4e7584a08b83661f2cec P 14aa376377e34cae9a387069ec429d4a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:08:59.432354 8442 raft_consensus.cc:686] T c54ecfa8868f4e7584a08b83661f2cec P 14aa376377e34cae9a387069ec429d4a [term 1 LEADER]: Becoming Leader. State: Replica: 14aa376377e34cae9a387069ec429d4a, State: Running, Role: LEADER
I20260924 04:08:59.432518 8442 consensus_queue.cc:237] T c54ecfa8868f4e7584a08b83661f2cec P 14aa376377e34cae9a387069ec429d4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc944248f10b48bc98c503b0f733a0e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45551 } } peers { permanent_uuid: "61447010247c4936bc9a9fcb2af3d202" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45391 } } peers { permanent_uuid: "14aa376377e34cae9a387069ec429d4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42625 } }
I20260924 04:08:59.434638 7926 catalog_manager.cc:5495] T c54ecfa8868f4e7584a08b83661f2cec P 14aa376377e34cae9a387069ec429d4a reported cstate change: term changed from 0 to 1, leader changed from <none> to 14aa376377e34cae9a387069ec429d4a (127.0.98.3). New cstate: current_term: 1 leader_uuid: "14aa376377e34cae9a387069ec429d4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc944248f10b48bc98c503b0f733a0e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "61447010247c4936bc9a9fcb2af3d202" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14aa376377e34cae9a387069ec429d4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42625 } health_report { overall_health: HEALTHY } } }
I20260924 04:08:59.465670 8433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:59.462747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60564 (local address 127.0.98.3:42625)
0924 04:08:59.462853 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0924 04:08:59.462859 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:08:59.462910 (+ 51us) server_negotiation.cc:407] Connection header received
0924 04:08:59.463019 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:08:59.463021 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:08:59.463064 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:08:59.463125 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:08:59.463303 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:59.463789 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:59.464558 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:08:59.464855 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:08:59.464976 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:08:59.465259 (+ 283us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:08:59.465325 (+ 66us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:08:59.465377 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 04:08:59.465494 (+ 117us) server_negotiation.cc:299] Negotiation successful
0924 04:08:59.465537 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 04:08:59.471732 8082 raft_consensus.cc:1261] T c54ecfa8868f4e7584a08b83661f2cec P cc944248f10b48bc98c503b0f733a0e9 [term 1 FOLLOWER]: Refusing update from remote peer 14aa376377e34cae9a387069ec429d4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:08:59.471700 8223 raft_consensus.cc:1261] T c54ecfa8868f4e7584a08b83661f2cec P 61447010247c4936bc9a9fcb2af3d202 [term 1 FOLLOWER]: Refusing update from remote peer 14aa376377e34cae9a387069ec429d4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:08:59.472124 8442 consensus_queue.cc:1035] T c54ecfa8868f4e7584a08b83661f2cec P 14aa376377e34cae9a387069ec429d4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc944248f10b48bc98c503b0f733a0e9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:08:59.472325 8491 consensus_queue.cc:1035] T c54ecfa8868f4e7584a08b83661f2cec P 14aa376377e34cae9a387069ec429d4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "61447010247c4936bc9a9fcb2af3d202" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:08:59.505034 7926 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:08:59.505174 7926 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:08:59.506956 7926 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 37584f0cfe7049d291a195adfd8d7aaf: Sending DeleteTablet for 3 replicas of tablet c54ecfa8868f4e7584a08b83661f2cec
I20260924 04:08:59.507511 8343 tablet_service.cc:1514] Processing DeleteTablet for tablet c54ecfa8868f4e7584a08b83661f2cec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:08:59 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57816
I20260924 04:08:59.507521 8063 tablet_service.cc:1514] Processing DeleteTablet for tablet c54ecfa8868f4e7584a08b83661f2cec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:08:59 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38806
I20260924 04:08:59.507690 8459 tablet_replica.cc:330] T c54ecfa8868f4e7584a08b83661f2cec P 14aa376377e34cae9a387069ec429d4a: stopping tablet replica
I20260924 04:08:59.507725 8461 tablet_replica.cc:330] T c54ecfa8868f4e7584a08b83661f2cec P cc944248f10b48bc98c503b0f733a0e9: stopping tablet replica
I20260924 04:08:59.507804 8459 raft_consensus.cc:2229] T c54ecfa8868f4e7584a08b83661f2cec P 14aa376377e34cae9a387069ec429d4a [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:08:59.507835 8461 raft_consensus.cc:2229] T c54ecfa8868f4e7584a08b83661f2cec P cc944248f10b48bc98c503b0f733a0e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:08:59.507944 8461 raft_consensus.cc:2258] T c54ecfa8868f4e7584a08b83661f2cec P cc944248f10b48bc98c503b0f733a0e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:08:59.507989 8459 raft_consensus.cc:2258] T c54ecfa8868f4e7584a08b83661f2cec P 14aa376377e34cae9a387069ec429d4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:08:59.508237 8203 tablet_service.cc:1514] Processing DeleteTablet for tablet c54ecfa8868f4e7584a08b83661f2cec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:08:59 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35930
I20260924 04:08:59.508291 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 7998
I20260924 04:08:59.508431 8460 tablet_replica.cc:330] T c54ecfa8868f4e7584a08b83661f2cec P 61447010247c4936bc9a9fcb2af3d202: stopping tablet replica
I20260924 04:08:59.508574 8460 raft_consensus.cc:2229] T c54ecfa8868f4e7584a08b83661f2cec P 61447010247c4936bc9a9fcb2af3d202 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:08:59.508625 8459 ts_tablet_manager.cc:1905] T c54ecfa8868f4e7584a08b83661f2cec P 14aa376377e34cae9a387069ec429d4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:08:59.508714 8460 raft_consensus.cc:2258] T c54ecfa8868f4e7584a08b83661f2cec P 61447010247c4936bc9a9fcb2af3d202 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:08:59.510377 8460 ts_tablet_manager.cc:1905] T c54ecfa8868f4e7584a08b83661f2cec P 61447010247c4936bc9a9fcb2af3d202: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:08:59.511111 8459 ts_tablet_manager.cc:1918] T c54ecfa8868f4e7584a08b83661f2cec P 14aa376377e34cae9a387069ec429d4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 04:08:59.511195 8459 log.cc:1196] T c54ecfa8868f4e7584a08b83661f2cec P 14aa376377e34cae9a387069ec429d4a: Deleting WAL directory at /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/minicluster-data/ts-2/wal/wals/c54ecfa8868f4e7584a08b83661f2cec
I20260924 04:08:59.511479 8459 ts_tablet_manager.cc:1939] T c54ecfa8868f4e7584a08b83661f2cec P 14aa376377e34cae9a387069ec429d4a: Deleting consensus metadata
I20260924 04:08:59.512445 7910 catalog_manager.cc:4841] TS 14aa376377e34cae9a387069ec429d4a (127.0.98.3:42625): tablet c54ecfa8868f4e7584a08b83661f2cec (table test-table [id=c1f6791e039548ecafc01e94860b32b0]) successfully deleted
I20260924 04:08:59.512715 8460 ts_tablet_manager.cc:1918] T c54ecfa8868f4e7584a08b83661f2cec P 61447010247c4936bc9a9fcb2af3d202: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 04:08:59.512787 8460 log.cc:1196] T c54ecfa8868f4e7584a08b83661f2cec P 61447010247c4936bc9a9fcb2af3d202: Deleting WAL directory at /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790222898807720-392-0/minicluster-data/ts-1/wal/wals/c54ecfa8868f4e7584a08b83661f2cec
I20260924 04:08:59.513093 8460 ts_tablet_manager.cc:1939] T c54ecfa8868f4e7584a08b83661f2cec P 61447010247c4936bc9a9fcb2af3d202: Deleting consensus metadata
I20260924 04:08:59.513700 7911 catalog_manager.cc:4841] TS 61447010247c4936bc9a9fcb2af3d202 (127.0.98.2:45391): tablet c54ecfa8868f4e7584a08b83661f2cec (table test-table [id=c1f6791e039548ecafc01e94860b32b0]) successfully deleted
W20260924 04:08:59.516067 7911 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45551 (error 108)
I20260924 04:08:59.516819 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 8138
I20260924 04:08:59.517783 8520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:08:59.516853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45551 (local address 127.0.0.1:57676)
0924 04:08:59.517593 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0924 04:08:59.517678 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":648,"thread_start_us":71,"threads_started":1}
W20260924 04:08:59.517905 7911 catalog_manager.cc:4584] TS cc944248f10b48bc98c503b0f733a0e9 (127.0.98.1:45551): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c54ecfa8868f4e7584a08b83661f2cec: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45551: connect: Connection refused (error 111)
I20260924 04:08:59.524739 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 8278
I20260924 04:08:59.531327 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 7894
2026-09-24T04:08:59Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2931 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790222898807720-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:08:59 dist-test-slave-00ld krb5kdc[8523](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:08:59 dist-test-slave-00ld krb5kdc[8523](info): set up 2 sockets
Sep 24 04:08:59 dist-test-slave-00ld krb5kdc[8523](info): commencing operation
krb5kdc: starting...
W20260924 04:09:00.581363 392 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790222898807720-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790222898807720-392-0/krb5kdc/test-user.keytab.
Sep 24 04:09:00 dist-test-slave-00ld krb5kdc[8523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222940, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:09:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:09:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:09:00.701350 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790222898807720-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790222898807720-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790222898807720-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42597
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42597
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:09:00.812884 8536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:09:00.813162 8536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:09:00.813205 8536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:09:00.817354 8536 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:09:00.817417 8536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:09:00.817437 8536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:09:00.817454 8536 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:09:00.817503 8536 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:09:00.822862 8536 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790222898807720-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790222898807720-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42597
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42597
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8536
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790222898807720-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:09:00.824020 8536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:09:00.825157 8536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:09:00.833787 8544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:09:00.833799 8542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:09:00.833828 8541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:09:00.833837 8536 server_base.cc:1034] running on GCE node
I20260924 04:09:00.834677 8536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:09:00.835840 8536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:09:00.837009 8536 hybrid_clock.cc:648] HybridClock initialized: now 1790222940836980 us; error 48 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-24T04:09:00Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1316 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:09:00 dist-test-slave-00ld krb5kdc[8549](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:09:00 dist-test-slave-00ld krb5kdc[8549](info): set up 2 sockets
Sep 24 04:09:00 dist-test-slave-00ld krb5kdc[8549](info): commencing operation
krb5kdc: starting...
W20260924 04:09:01.889465 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/krb5kdc/test-user.keytab.
Sep 24 04:09:01 dist-test-slave-00ld krb5kdc[8549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222941, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:09:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:09:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:09:02.011396 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42249
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42249
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:09:02.120311 8562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:09:02.120637 8562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:09:02.120743 8562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:09:02.124768 8562 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:09:02.124867 8562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:09:02.124907 8562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:09:02.124961 8562 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:09:02.125010 8562 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:09:02.130532 8562 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42249
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42249
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8562
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:09:02.131639 8562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:09:02.132727 8562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:09:02.141549 8562 server_base.cc:1034] running on GCE node
W20260924 04:09:02.141486 8567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:09:02.141583 8568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:09:02.141487 8570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:09:02.142297 8562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:09:02.143410 8562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:09:02.144569 8562 hybrid_clock.cc:648] HybridClock initialized: now 1790222942144553 us; error 32 us; skew 500 ppm
Sep 24 04:09:02 dist-test-slave-00ld krb5kdc[8549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222942, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:09:02.147222 8562 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:09:02.148080 8562 webserver.cc:460] Webserver started at http://127.0.98.62:39469/ using document root <none> and password file <none>
I20260924 04:09:02.148633 8562 fs_manager.cc:359] Metadata directory not provided
I20260924 04:09:02.148738 8562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:09:02.148970 8562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:09:02.150983 8562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/master-0/data/instance:
uuid: "ad4bed1ef4ce484aa7f3414693af0526"
format_stamp: "Formatted at 2026-09-24 04:09:02 on dist-test-slave-00ld"
server_key: "a01d5628c84b1e780d612fc59fe58f7d"
server_key_iv: "f2c90e8f65fda5924401e2e5a9ab6864"
server_key_version: "encryptionkey@0"
I20260924 04:09:02.151539 8562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/master-0/wal/instance:
uuid: "ad4bed1ef4ce484aa7f3414693af0526"
format_stamp: "Formatted at 2026-09-24 04:09:02 on dist-test-slave-00ld"
server_key: "a01d5628c84b1e780d612fc59fe58f7d"
server_key_iv: "f2c90e8f65fda5924401e2e5a9ab6864"
server_key_version: "encryptionkey@0"
I20260924 04:09:02.155301 8562 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 04:09:02.157850 8577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:09:02.159104 8562 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:09:02.159245 8562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/master-0/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/master-0/wal
uuid: "ad4bed1ef4ce484aa7f3414693af0526"
format_stamp: "Formatted at 2026-09-24 04:09:02 on dist-test-slave-00ld"
server_key: "a01d5628c84b1e780d612fc59fe58f7d"
server_key_iv: "f2c90e8f65fda5924401e2e5a9ab6864"
server_key_version: "encryptionkey@0"
I20260924 04:09:02.159358 8562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:09:02.174870 8562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:09:02.177721 8562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:09:02.177922 8562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:09:02.184398 8562 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42249
I20260924 04:09:02.184464 8629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42249 every 8 connection(s)
I20260924 04:09:02.185344 8562 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:09:02.187436 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 8562
I20260924 04:09:02.187582 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/master-0/wal/instance
I20260924 04:09:02.187877 392 external_mini_cluster.cc:1428] Setting key 8a377c02e2613452274b05efb5cfa557
I20260924 04:09:02.188129 8630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:09:02.195241 8630 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad4bed1ef4ce484aa7f3414693af0526: Bootstrap starting.
Sep 24 04:09:02 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222941, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:09:02.197777 8630 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad4bed1ef4ce484aa7f3414693af0526: Neither blocks nor log segments found. Creating new log.
I20260924 04:09:02.198501 8630 log.cc:824] T 00000000000000000000000000000000 P ad4bed1ef4ce484aa7f3414693af0526: Log is configured to *not* fsync() on all Append() calls
I20260924 04:09:02.200659 8630 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad4bed1ef4ce484aa7f3414693af0526: No bootstrap required, opened a new log
I20260924 04:09:02.202611 8633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:09:02.189195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39098 (local address 127.0.98.62:42249)
0924 04:09:02.189611 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0924 04:09:02.189632 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0924 04:09:02.189676 (+ 44us) server_negotiation.cc:407] Connection header received
0924 04:09:02.190466 (+ 790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:09:02.190490 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:09:02.190926 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:09:02.191309 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:09:02.191974 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:02.193445 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:02.194276 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:02.194942 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:02.195097 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:02.197799 (+ 2702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:09:02.197804 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:09:02.197816 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:09:02.197863 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:09:02.199577 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:09:02.200067 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:09:02.200071 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:09:02.200076 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:09:02.200179 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:09:02.200470 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:09:02.200472 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:09:02.200473 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:09:02.200945 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:09:02.201092 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 04:09:02.201397 (+ 305us) server_negotiation.cc:299] Negotiation successful
0924 04:09:02.201693 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":107,"threads_started":1}
I20260924 04:09:02.204083 8630 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad4bed1ef4ce484aa7f3414693af0526 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad4bed1ef4ce484aa7f3414693af0526" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42249 } }
I20260924 04:09:02.204361 8630 raft_consensus.cc:374] T 00000000000000000000000000000000 P ad4bed1ef4ce484aa7f3414693af0526 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:09:02.204425 8630 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad4bed1ef4ce484aa7f3414693af0526 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad4bed1ef4ce484aa7f3414693af0526, State: Initialized, Role: FOLLOWER
I20260924 04:09:02.204874 8630 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad4bed1ef4ce484aa7f3414693af0526 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad4bed1ef4ce484aa7f3414693af0526" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42249 } }
I20260924 04:09:02.205031 8630 raft_consensus.cc:388] T 00000000000000000000000000000000 P ad4bed1ef4ce484aa7f3414693af0526 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:09:02.205111 8630 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad4bed1ef4ce484aa7f3414693af0526 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:09:02.205232 8630 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad4bed1ef4ce484aa7f3414693af0526 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:09:02.206118 8630 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad4bed1ef4ce484aa7f3414693af0526 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad4bed1ef4ce484aa7f3414693af0526" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42249 } }
I20260924 04:09:02.206487 8630 leader_election.cc:302] T 00000000000000000000000000000000 P ad4bed1ef4ce484aa7f3414693af0526 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad4bed1ef4ce484aa7f3414693af0526; no voters:
I20260924 04:09:02.206791 8630 leader_election.cc:288] T 00000000000000000000000000000000 P ad4bed1ef4ce484aa7f3414693af0526 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:09:02.206913 8635 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad4bed1ef4ce484aa7f3414693af0526 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:09:02.207150 8635 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad4bed1ef4ce484aa7f3414693af0526 [term 1 LEADER]: Becoming Leader. State: Replica: ad4bed1ef4ce484aa7f3414693af0526, State: Running, Role: LEADER
I20260924 04:09:02.207443 8635 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad4bed1ef4ce484aa7f3414693af0526 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad4bed1ef4ce484aa7f3414693af0526" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42249 } }
I20260924 04:09:02.207988 8630 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad4bed1ef4ce484aa7f3414693af0526 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:09:02.209105 8637 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad4bed1ef4ce484aa7f3414693af0526 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad4bed1ef4ce484aa7f3414693af0526. Latest consensus state: current_term: 1 leader_uuid: "ad4bed1ef4ce484aa7f3414693af0526" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad4bed1ef4ce484aa7f3414693af0526" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42249 } } }
I20260924 04:09:02.209268 8637 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad4bed1ef4ce484aa7f3414693af0526 [sys.catalog]: This master's current role is: LEADER
I20260924 04:09:02.209651 8644 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:09:02.209105 8636 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad4bed1ef4ce484aa7f3414693af0526 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ad4bed1ef4ce484aa7f3414693af0526" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad4bed1ef4ce484aa7f3414693af0526" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42249 } } }
I20260924 04:09:02.210107 8636 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad4bed1ef4ce484aa7f3414693af0526 [sys.catalog]: This master's current role is: LEADER
I20260924 04:09:02.212733 8644 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:09:02.217764 8644 catalog_manager.cc:1347] Generated new cluster ID: 96dd6d4d8791469893985cfa25d4c7cc
I20260924 04:09:02.217834 8644 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:09:02.234835 8644 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:09:02.236196 8644 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:09:02.242086 8644 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ad4bed1ef4ce484aa7f3414693af0526: Generated new TSK 0
I20260924 04:09:02.242833 8644 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:09:02.271528 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42249
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:09:02.383159 8656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:09:02.383451 8656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:09:02.383527 8656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:09:02.387559 8656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:09:02.387652 8656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:09:02.387742 8656 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:09:02.393435 8656 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42249
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8656
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:09:02.394510 8656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:09:02.395541 8656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:09:02.404808 8664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:09:02.404814 8661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:09:02.404910 8656 server_base.cc:1034] running on GCE node
W20260924 04:09:02.405136 8662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:09:02.405514 8656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:09:02.406180 8656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:09:02.407347 8656 hybrid_clock.cc:648] HybridClock initialized: now 1790222942407332 us; error 26 us; skew 500 ppm
Sep 24 04:09:02 dist-test-slave-00ld krb5kdc[8549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222942, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:09:02.410156 8656 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:09:02.410986 8656 webserver.cc:460] Webserver started at http://127.0.98.1:38285/ using document root <none> and password file <none>
I20260924 04:09:02.411487 8656 fs_manager.cc:359] Metadata directory not provided
I20260924 04:09:02.411543 8656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:09:02.411698 8656 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:09:02.413655 8656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/ts-0/data/instance:
uuid: "265ca1967fe64eb4bc4aa48df5d2c6e8"
format_stamp: "Formatted at 2026-09-24 04:09:02 on dist-test-slave-00ld"
server_key: "beb0263e1452c7168dfd84283fea2476"
server_key_iv: "5ce66e913a7f5a4bfee375f0661d291f"
server_key_version: "encryptionkey@0"
I20260924 04:09:02.414149 8656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/ts-0/wal/instance:
uuid: "265ca1967fe64eb4bc4aa48df5d2c6e8"
format_stamp: "Formatted at 2026-09-24 04:09:02 on dist-test-slave-00ld"
server_key: "beb0263e1452c7168dfd84283fea2476"
server_key_iv: "5ce66e913a7f5a4bfee375f0661d291f"
server_key_version: "encryptionkey@0"
I20260924 04:09:02.417660 8656 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260924 04:09:02.419814 8671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:09:02.420990 8656 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 04:09:02.421106 8656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/ts-0/wal
uuid: "265ca1967fe64eb4bc4aa48df5d2c6e8"
format_stamp: "Formatted at 2026-09-24 04:09:02 on dist-test-slave-00ld"
server_key: "beb0263e1452c7168dfd84283fea2476"
server_key_iv: "5ce66e913a7f5a4bfee375f0661d291f"
server_key_version: "encryptionkey@0"
I20260924 04:09:02.421206 8656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:09:02.432972 8656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:09:02.435837 8656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:09:02.436033 8656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:09:02.436606 8656 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:09:02.437438 8656 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:09:02.437502 8656 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:09:02.437569 8656 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:09:02.437614 8656 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:09:02.446448 8656 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43929
I20260924 04:09:02.446465 8784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43929 every 8 connection(s)
I20260924 04:09:02.447331 8656 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:09:02.447878 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 8656
I20260924 04:09:02.448024 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:09:02.448308 392 external_mini_cluster.cc:1428] Setting key 949a0c143e78ed3ca7d7ae0215c00e5c
Sep 24 04:09:02 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222942, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 04:09:02.462917 8633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:09:02.449246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57703 (local address 127.0.98.62:42249)
0924 04:09:02.449370 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 04:09:02.449374 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:09:02.450238 (+ 864us) server_negotiation.cc:407] Connection header received
0924 04:09:02.451797 (+ 1559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:09:02.451801 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:09:02.451845 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:09:02.451995 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:09:02.453329 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:02.453913 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:02.455763 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:02.456005 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:02.456096 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:02.459071 (+ 2975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:09:02.459075 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:09:02.459078 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:09:02.459108 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:09:02.460479 (+ 1371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:09:02.461150 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:09:02.461153 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:09:02.461154 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:09:02.461226 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:09:02.461603 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:09:02.461609 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:09:02.461614 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:09:02.461871 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:09:02.461960 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 04:09:02.462633 (+ 673us) server_negotiation.cc:299] Negotiation successful
0924 04:09:02.462764 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 04:09:02.464051 8787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:09:02.449650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42249 (local address 127.0.98.1:57703)
0924 04:09:02.450109 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0924 04:09:02.450150 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 04:09:02.451541 (+ 1391us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:09:02.452176 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:09:02.452187 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:09:02.452665 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:09:02.453191 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:02.453204 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:02.454036 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:02.454042 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:02.455620 (+ 1578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:02.455634 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:02.456165 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:02.456172 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:02.456421 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:02.457129 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:09:02.457149 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:09:02.458876 (+ 1727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:09:02.460587 (+ 1711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:09:02.460592 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:09:02.460605 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:09:02.461062 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:09:02.461312 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:09:02.461314 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:09:02.461315 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:09:02.461501 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:09:02.461985 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:09:02.461992 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:09:02.462351 (+ 359us) client_negotiation.cc:769] Sending connection context
0924 04:09:02.462571 (+ 220us) client_negotiation.cc:240] Negotiation successful
0924 04:09:02.463013 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":143,"threads_started":1}
I20260924 04:09:02.465301 8785 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42249
I20260924 04:09:02.465559 8785 heartbeater.cc:464] Registering TS with master...
I20260924 04:09:02.466096 8785 heartbeater.cc:511] Master 127.0.98.62:42249 requested a full tablet report, sending...
I20260924 04:09:02.467674 8594 ts_manager.cc:194] Registered new tserver with Master: 265ca1967fe64eb4bc4aa48df5d2c6e8 (127.0.98.1:43929)
I20260924 04:09:02.468881 8594 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57703
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:09:02.479519 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42249
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:09:02.587886 8790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:09:02.588212 8790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:09:02.588292 8790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:09:02.592447 8790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:09:02.592549 8790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:09:02.592643 8790 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:09:02.598075 8790 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42249
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8790
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:09:02.599187 8790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:09:02.600221 8790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:09:02.610291 8798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:09:02.610307 8796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:09:02.610303 8795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:09:02.610565 8790 server_base.cc:1034] running on GCE node
I20260924 04:09:02.610991 8790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:09:02.611685 8790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:09:02.612891 8790 hybrid_clock.cc:648] HybridClock initialized: now 1790222942612874 us; error 34 us; skew 500 ppm
Sep 24 04:09:02 dist-test-slave-00ld krb5kdc[8549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222942, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:09:02.615829 8790 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:09:02.616881 8790 webserver.cc:460] Webserver started at http://127.0.98.2:44703/ using document root <none> and password file <none>
I20260924 04:09:02.617473 8790 fs_manager.cc:359] Metadata directory not provided
I20260924 04:09:02.617561 8790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:09:02.617806 8790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:09:02.619963 8790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/ts-1/data/instance:
uuid: "619e4d856d2446f39d05822025ec3db6"
format_stamp: "Formatted at 2026-09-24 04:09:02 on dist-test-slave-00ld"
server_key: "48ae6b102ef490043e165b5b074117b7"
server_key_iv: "316d863b23026a6c15ddddaeaac5f011"
server_key_version: "encryptionkey@0"
I20260924 04:09:02.620584 8790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/ts-1/wal/instance:
uuid: "619e4d856d2446f39d05822025ec3db6"
format_stamp: "Formatted at 2026-09-24 04:09:02 on dist-test-slave-00ld"
server_key: "48ae6b102ef490043e165b5b074117b7"
server_key_iv: "316d863b23026a6c15ddddaeaac5f011"
server_key_version: "encryptionkey@0"
I20260924 04:09:02.624927 8790 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 04:09:02.627424 8805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:09:02.628777 8790 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:09:02.628894 8790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/ts-1/wal
uuid: "619e4d856d2446f39d05822025ec3db6"
format_stamp: "Formatted at 2026-09-24 04:09:02 on dist-test-slave-00ld"
server_key: "48ae6b102ef490043e165b5b074117b7"
server_key_iv: "316d863b23026a6c15ddddaeaac5f011"
server_key_version: "encryptionkey@0"
I20260924 04:09:02.628998 8790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:09:02.642921 8790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:09:02.645797 8790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:09:02.645990 8790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:09:02.646567 8790 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:09:02.647375 8790 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:09:02.647421 8790 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:09:02.647485 8790 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:09:02.647522 8790 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:09:02.656448 8790 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33257
I20260924 04:09:02.656466 8918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33257 every 8 connection(s)
I20260924 04:09:02.657437 8790 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:09:02.665851 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 8790
I20260924 04:09:02.666013 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:09:02.666365 392 external_mini_cluster.cc:1428] Setting key 6284413a04deba2e143c71712d6b3d9d
Sep 24 04:09:02 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222942, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:09:02.672398 8633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:09:02.659350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45831 (local address 127.0.98.62:42249)
0924 04:09:02.659489 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 04:09:02.659493 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:09:02.660374 (+ 881us) server_negotiation.cc:407] Connection header received
0924 04:09:02.661830 (+ 1456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:09:02.661834 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:09:02.661898 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:09:02.661998 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:09:02.663234 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:02.663773 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:02.664959 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:02.665183 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:02.665290 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:02.668216 (+ 2926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:09:02.668219 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:09:02.668221 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:09:02.668244 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:09:02.670035 (+ 1791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:09:02.670551 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:09:02.670556 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:09:02.670559 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:09:02.670656 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:09:02.671084 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:09:02.671086 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:09:02.671087 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:09:02.671325 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:09:02.671405 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 04:09:02.672095 (+ 690us) server_negotiation.cc:299] Negotiation successful
0924 04:09:02.672276 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 04:09:02.673729 8921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:09:02.659781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42249 (local address 127.0.98.2:45831)
0924 04:09:02.660225 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0924 04:09:02.660275 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 04:09:02.661607 (+ 1332us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:09:02.662171 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:09:02.662179 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:09:02.662557 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:09:02.663094 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:02.663108 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:02.663892 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:02.663896 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:02.664845 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:02.664854 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:02.665331 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:02.665335 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:02.665503 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:02.666332 (+ 829us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:09:02.666353 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:09:02.668057 (+ 1704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:09:02.670180 (+ 2123us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:09:02.670185 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:09:02.670201 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:09:02.670457 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:09:02.670774 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:09:02.670780 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:09:02.670785 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:09:02.670981 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:09:02.671429 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:09:02.671439 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:09:02.671809 (+ 370us) client_negotiation.cc:769] Sending connection context
0924 04:09:02.672035 (+ 226us) client_negotiation.cc:240] Negotiation successful
0924 04:09:02.672442 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":118,"threads_started":1}
I20260924 04:09:02.675066 8919 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42249
I20260924 04:09:02.675324 8919 heartbeater.cc:464] Registering TS with master...
I20260924 04:09:02.675823 8919 heartbeater.cc:511] Master 127.0.98.62:42249 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 04:09:02.676898 8594 ts_manager.cc:194] Registered new tserver with Master: 619e4d856d2446f39d05822025ec3db6 (127.0.98.2:33257)
I20260924 04:09:02.677536 8594 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:45831
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:09:02.695487 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42249
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:09:02.806347 8924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:09:02.806623 8924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:09:02.806694 8924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:09:02.810974 8924 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:09:02.811056 8924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:09:02.811143 8924 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:09:02.816498 8924 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42249
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8924
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:09:02.817665 8924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:09:02.818679 8924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:09:02.828132 8930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:09:02.828115 8929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:09:02.828253 8932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:09:02.828537 8924 server_base.cc:1034] running on GCE node
I20260924 04:09:02.828940 8924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:09:02.829558 8924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:09:02.830729 8924 hybrid_clock.cc:648] HybridClock initialized: now 1790222942830716 us; error 28 us; skew 500 ppm
Sep 24 04:09:02 dist-test-slave-00ld krb5kdc[8549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222942, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:09:02.833492 8924 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:09:02.834373 8924 webserver.cc:460] Webserver started at http://127.0.98.3:43837/ using document root <none> and password file <none>
I20260924 04:09:02.834916 8924 fs_manager.cc:359] Metadata directory not provided
I20260924 04:09:02.834976 8924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:09:02.835188 8924 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:09:02.837256 8924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/ts-2/data/instance:
uuid: "0d6708cb99a94cc7ba024d14292d4e85"
format_stamp: "Formatted at 2026-09-24 04:09:02 on dist-test-slave-00ld"
server_key: "2fb5212dcf9c7ca4463b6d8cfd34d098"
server_key_iv: "40cd90bb63125b061b3f639d836add25"
server_key_version: "encryptionkey@0"
I20260924 04:09:02.837836 8924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/ts-2/wal/instance:
uuid: "0d6708cb99a94cc7ba024d14292d4e85"
format_stamp: "Formatted at 2026-09-24 04:09:02 on dist-test-slave-00ld"
server_key: "2fb5212dcf9c7ca4463b6d8cfd34d098"
server_key_iv: "40cd90bb63125b061b3f639d836add25"
server_key_version: "encryptionkey@0"
I20260924 04:09:02.841997 8924 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 04:09:02.844501 8939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:09:02.846068 8924 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 04:09:02.846225 8924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/ts-2/wal
uuid: "0d6708cb99a94cc7ba024d14292d4e85"
format_stamp: "Formatted at 2026-09-24 04:09:02 on dist-test-slave-00ld"
server_key: "2fb5212dcf9c7ca4463b6d8cfd34d098"
server_key_iv: "40cd90bb63125b061b3f639d836add25"
server_key_version: "encryptionkey@0"
I20260924 04:09:02.846344 8924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:09:02.879385 8924 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:09:02.882508 8924 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:09:02.882781 8924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:09:02.883421 8924 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:09:02.884270 8924 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:09:02.884320 8924 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:09:02.884392 8924 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:09:02.884429 8924 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:09:02.893918 8924 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:32865
I20260924 04:09:02.893937 9052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:32865 every 8 connection(s)
I20260924 04:09:02.894870 8924 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:09:02.895606 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 8924
I20260924 04:09:02.895742 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:09:02.896029 392 external_mini_cluster.cc:1428] Setting key 059f0b07e5b6568e6c1147a6d71efab2
Sep 24 04:09:02 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222942, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:09:02.909515 8633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:09:02.896853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37011 (local address 127.0.98.62:42249)
0924 04:09:02.897009 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 04:09:02.897015 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:09:02.898042 (+ 1027us) server_negotiation.cc:407] Connection header received
0924 04:09:02.899757 (+ 1715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:09:02.899761 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:09:02.899817 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:09:02.899894 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:09:02.901105 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:02.901647 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:02.902805 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:02.903045 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:02.903151 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:02.905646 (+ 2495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:09:02.905650 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:09:02.905653 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:09:02.905682 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:09:02.906995 (+ 1313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:09:02.907622 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:09:02.907625 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:09:02.907627 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:09:02.907728 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:09:02.908082 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:09:02.908086 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:09:02.908088 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:09:02.908328 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:09:02.908476 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 04:09:02.909199 (+ 723us) server_negotiation.cc:299] Negotiation successful
0924 04:09:02.909328 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 04:09:02.910523 9055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:09:02.897391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42249 (local address 127.0.98.3:37011)
0924 04:09:02.897901 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0924 04:09:02.897942 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 04:09:02.899471 (+ 1529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:09:02.900073 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:09:02.900080 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:09:02.900438 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:09:02.900952 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:02.900965 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:02.901768 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:02.901772 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:02.902683 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:02.902692 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:02.903189 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:02.903193 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:02.903359 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:02.903887 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:09:02.903903 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:09:02.905481 (+ 1578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:09:02.907127 (+ 1646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:09:02.907134 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:09:02.907150 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:09:02.907481 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:09:02.907824 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:09:02.907829 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:09:02.907832 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:09:02.907978 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:09:02.908499 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:09:02.908506 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:09:02.908912 (+ 406us) client_negotiation.cc:769] Sending connection context
0924 04:09:02.909157 (+ 245us) client_negotiation.cc:240] Negotiation successful
0924 04:09:02.909525 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":158,"threads_started":1}
I20260924 04:09:02.911542 9053 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42249
I20260924 04:09:02.911806 9053 heartbeater.cc:464] Registering TS with master...
I20260924 04:09:02.912305 9053 heartbeater.cc:511] Master 127.0.98.62:42249 requested a full tablet report, sending...
I20260924 04:09:02.913312 8594 ts_manager.cc:194] Registered new tserver with Master: 0d6708cb99a94cc7ba024d14292d4e85 (127.0.98.3:32865)
I20260924 04:09:02.913872 8594 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:37011
I20260924 04:09:02.921965 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260924 04:09:02.996332 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 8656
I20260924 04:09:03.001933 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 8790
I20260924 04:09:03.007085 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 8924
I20260924 04:09:03.012542 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 8562
I20260924 04:09:03.018798 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42249
--webserver_interface=127.0.98.62
--webserver_port=39469
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42249
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:09:03.129803 9062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:09:03.130088 9062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:09:03.130160 9062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:09:03.134081 9062 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:09:03.134163 9062 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:09:03.134202 9062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:09:03.134255 9062 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:09:03.134294 9062 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:09:03.139506 9062 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42249
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42249
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=39469
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9062
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:09:03.140604 9062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:09:03.141662 9062 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:09:03.149967 9062 server_base.cc:1034] running on GCE node
W20260924 04:09:03.149856 9068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:09:03.149868 9070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:09:03.149856 9067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:09:03.150739 9062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:09:03.151763 9062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:09:03.152923 9062 hybrid_clock.cc:648] HybridClock initialized: now 1790222943152906 us; error 27 us; skew 500 ppm
Sep 24 04:09:03 dist-test-slave-00ld krb5kdc[8549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222943, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:09:03.155730 9062 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:09:03.156635 9062 webserver.cc:460] Webserver started at http://127.0.98.62:39469/ using document root <none> and password file <none>
I20260924 04:09:03.157223 9062 fs_manager.cc:359] Metadata directory not provided
I20260924 04:09:03.157299 9062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:09:03.161132 9062 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:09:03.163261 9077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:09:03.164592 9062 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:09:03.164803 9062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/master-0/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/master-0/wal
uuid: "ad4bed1ef4ce484aa7f3414693af0526"
format_stamp: "Formatted at 2026-09-24 04:09:02 on dist-test-slave-00ld"
server_key: "a01d5628c84b1e780d612fc59fe58f7d"
server_key_iv: "f2c90e8f65fda5924401e2e5a9ab6864"
server_key_version: "encryptionkey@0"
I20260924 04:09:03.165350 9062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:09:03.183333 9062 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:09:03.186225 9062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:09:03.186375 9062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:09:03.193295 9062 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42249
I20260924 04:09:03.193313 9129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42249 every 8 connection(s)
I20260924 04:09:03.194386 9062 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:09:03.194636 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 9062
I20260924 04:09:03.195156 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:43929
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:42249
--webserver_port=38285
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:09:03.198818 9130 sys_catalog.cc:263] Verifying existing consensus state
I20260924 04:09:03.200418 9130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad4bed1ef4ce484aa7f3414693af0526: Bootstrap starting.
I20260924 04:09:03.216532 9130 log.cc:824] T 00000000000000000000000000000000 P ad4bed1ef4ce484aa7f3414693af0526: Log is configured to *not* fsync() on all Append() calls
I20260924 04:09:03.222782 9130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad4bed1ef4ce484aa7f3414693af0526: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 04:09:03.223146 9130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad4bed1ef4ce484aa7f3414693af0526: Bootstrap complete.
I20260924 04:09:03.227229 9130 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad4bed1ef4ce484aa7f3414693af0526 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad4bed1ef4ce484aa7f3414693af0526" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42249 } }
I20260924 04:09:03.227635 9130 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad4bed1ef4ce484aa7f3414693af0526 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad4bed1ef4ce484aa7f3414693af0526, State: Initialized, Role: FOLLOWER
I20260924 04:09:03.228096 9130 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad4bed1ef4ce484aa7f3414693af0526 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad4bed1ef4ce484aa7f3414693af0526" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42249 } }
I20260924 04:09:03.228245 9130 raft_consensus.cc:388] T 00000000000000000000000000000000 P ad4bed1ef4ce484aa7f3414693af0526 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:09:03.228317 9130 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad4bed1ef4ce484aa7f3414693af0526 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:09:03.228420 9130 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad4bed1ef4ce484aa7f3414693af0526 [term 1 FOLLOWER]: Advancing to term 2
I20260924 04:09:03.230324 9130 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad4bed1ef4ce484aa7f3414693af0526 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad4bed1ef4ce484aa7f3414693af0526" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42249 } }
I20260924 04:09:03.230640 9130 leader_election.cc:302] T 00000000000000000000000000000000 P ad4bed1ef4ce484aa7f3414693af0526 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad4bed1ef4ce484aa7f3414693af0526; no voters:
I20260924 04:09:03.230947 9130 leader_election.cc:288] T 00000000000000000000000000000000 P ad4bed1ef4ce484aa7f3414693af0526 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 04:09:03.231032 9134 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad4bed1ef4ce484aa7f3414693af0526 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 04:09:03.231247 9134 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad4bed1ef4ce484aa7f3414693af0526 [term 2 LEADER]: Becoming Leader. State: Replica: ad4bed1ef4ce484aa7f3414693af0526, State: Running, Role: LEADER
I20260924 04:09:03.231588 9134 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad4bed1ef4ce484aa7f3414693af0526 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad4bed1ef4ce484aa7f3414693af0526" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42249 } }
I20260924 04:09:03.232136 9130 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad4bed1ef4ce484aa7f3414693af0526 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:09:03.233105 9135 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad4bed1ef4ce484aa7f3414693af0526 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ad4bed1ef4ce484aa7f3414693af0526" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad4bed1ef4ce484aa7f3414693af0526" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42249 } } }
I20260924 04:09:03.233112 9136 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad4bed1ef4ce484aa7f3414693af0526 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad4bed1ef4ce484aa7f3414693af0526. Latest consensus state: current_term: 2 leader_uuid: "ad4bed1ef4ce484aa7f3414693af0526" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad4bed1ef4ce484aa7f3414693af0526" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42249 } } }
I20260924 04:09:03.233246 9135 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad4bed1ef4ce484aa7f3414693af0526 [sys.catalog]: This master's current role is: LEADER
I20260924 04:09:03.233254 9136 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad4bed1ef4ce484aa7f3414693af0526 [sys.catalog]: This master's current role is: LEADER
I20260924 04:09:03.233793 9143 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:09:03.236984 9143 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:09:03.237624 9143 catalog_manager.cc:1259] Loaded cluster ID: 96dd6d4d8791469893985cfa25d4c7cc
I20260924 04:09:03.237682 9143 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:09:03.239326 9143 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:09:03.239888 9143 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ad4bed1ef4ce484aa7f3414693af0526: Loaded TSK: 0
I20260924 04:09:03.240967 9143 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 04:09:03.318924 9132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:09:03.319226 9132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:09:03.319306 9132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:09:03.323359 9132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:09:03.323450 9132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:09:03.323537 9132 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:09:03.329095 9132 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:43929
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=38285
--tserver_master_addrs=127.0.98.62:42249
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9132
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:09:03.330137 9132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:09:03.331131 9132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:09:03.342235 9156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:09:03.344553 9157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:09:03.345317 9159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:09:03.346730 9132 server_base.cc:1034] running on GCE node
I20260924 04:09:03.347707 9132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:09:03.348542 9132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:09:03.350049 9132 hybrid_clock.cc:648] HybridClock initialized: now 1790222943349996 us; error 59 us; skew 500 ppm
Sep 24 04:09:03 dist-test-slave-00ld krb5kdc[8549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222943, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:09:03.369463 9132 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260924 04:09:03.370783 9132 webserver.cc:460] Webserver started at http://127.0.98.1:38285/ using document root <none> and password file <none>
I20260924 04:09:03.371497 9132 fs_manager.cc:359] Metadata directory not provided
I20260924 04:09:03.371596 9132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:09:03.376415 9132 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 04:09:03.378901 9166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:09:03.380450 9132 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 04:09:03.380578 9132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/ts-0/wal
uuid: "265ca1967fe64eb4bc4aa48df5d2c6e8"
format_stamp: "Formatted at 2026-09-24 04:09:02 on dist-test-slave-00ld"
server_key: "beb0263e1452c7168dfd84283fea2476"
server_key_iv: "5ce66e913a7f5a4bfee375f0661d291f"
server_key_version: "encryptionkey@0"
I20260924 04:09:03.381317 9132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:09:03.400472 9132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:09:03.403852 9132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:09:03.404000 9132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:09:03.404524 9132 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:09:03.405323 9132 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:09:03.405367 9132 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:09:03.405406 9132 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:09:03.405424 9132 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:09:03.414479 9132 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43929
I20260924 04:09:03.414501 9279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43929 every 8 connection(s)
I20260924 04:09:03.415462 9132 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:09:03.421890 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 9132
I20260924 04:09:03.422387 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:33257
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:42249
--webserver_port=44703
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 04:09:03 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:09:03 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:09:03.429066 9283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:09:03.417736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42249 (local address 127.0.98.1:33599)
0924 04:09:03.418219 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0924 04:09:03.418277 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 04:09:03.419594 (+ 1317us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:09:03.420730 (+ 1136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:09:03.420739 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:09:03.421163 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:09:03.421620 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:03.421632 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:03.423037 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:03.423041 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:03.423901 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:03.423910 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:03.424648 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:03.424652 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:03.424883 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:03.425545 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:09:03.425571 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:09:03.428256 (+ 2685us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42249: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":117,"threads_started":1}
W20260924 04:09:03.429513 9280 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42249 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:42249: Client connection negotiation failed: client connection to 127.0.98.62:42249: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
W20260924 04:09:03.429589 9282 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:09:03.417628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33599 (local address 127.0.98.62:42249)
0924 04:09:03.418157 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0924 04:09:03.418177 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 04:09:03.418389 (+ 212us) server_negotiation.cc:407] Connection header received
0924 04:09:03.419998 (+ 1609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:09:03.420009 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:09:03.420299 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:09:03.420627 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:09:03.421789 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:03.422911 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:03.424023 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:03.424482 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:03.424663 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:03.428585 (+ 3922us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:33599: BlockingRecv error: recv got EOF from 127.0.98.1:33599 (error 108)
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":118,"threads_started":1}
Sep 24 04:09:03 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:09:03 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:09:03.435478 9283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:09:03.430059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42249 (local address 127.0.98.1:57379)
0924 04:09:03.430223 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 04:09:03.430235 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:09:03.430293 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:09:03.430565 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:09:03.430567 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:09:03.430691 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:09:03.430776 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:03.430780 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:03.431862 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:03.431865 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:03.432452 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:03.432460 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:03.433157 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:03.433160 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:03.433237 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:03.433712 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:09:03.433723 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:09:03.435390 (+ 1667us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42249: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":96}
W20260924 04:09:03.435649 9282 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:09:03.430110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57379 (local address 127.0.98.62:42249)
0924 04:09:03.430245 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 04:09:03.430251 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:09:03.430270 (+ 19us) server_negotiation.cc:407] Connection header received
0924 04:09:03.430381 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:09:03.430387 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:09:03.430450 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:09:03.430562 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:09:03.430877 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:03.431760 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:03.432570 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:03.433046 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:03.433177 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:03.435542 (+ 2365us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:57379: BlockingRecv error: recv got EOF from 127.0.98.1:57379 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 24 04:09:03 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:09:03 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:09:03.440902 9283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:09:03.435954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42249 (local address 127.0.98.1:56009)
0924 04:09:03.436089 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 04:09:03.436104 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:09:03.436164 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:09:03.436387 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:09:03.436389 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:09:03.436527 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:09:03.436588 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:03.436592 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:03.437561 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:03.437564 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:03.438275 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:03.438283 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:03.438806 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:03.438808 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:03.438859 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:03.439300 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:09:03.439311 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:09:03.440816 (+ 1505us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42249: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":75}
W20260924 04:09:03.441058 9282 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:09:03.436028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56009 (local address 127.0.98.62:42249)
0924 04:09:03.436121 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0924 04:09:03.436125 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:09:03.436139 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:09:03.436247 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:09:03.436250 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:09:03.436291 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:09:03.436369 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:09:03.436675 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:03.437461 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:03.438378 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:03.438718 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:03.438811 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:03.440955 (+ 2144us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:56009: BlockingRecv error: recv got EOF from 127.0.98.1:56009 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260924 04:09:03.441252 9280 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 04:09:03.536187 9284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:09:03.536454 9284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:09:03.536549 9284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:09:03.540634 9284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:09:03.540757 9284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:09:03.540881 9284 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:09:03.546082 9284 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:33257
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=44703
--tserver_master_addrs=127.0.98.62:42249
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9284
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:09:03.547111 9284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:09:03.548076 9284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:09:03.557263 9289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:09:03.557293 9290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:09:03.557412 9284 server_base.cc:1034] running on GCE node
W20260924 04:09:03.557583 9292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:09:03.557981 9284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:09:03.558610 9284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:09:03.559767 9284 hybrid_clock.cc:648] HybridClock initialized: now 1790222943559733 us; error 49 us; skew 500 ppm
Sep 24 04:09:03 dist-test-slave-00ld krb5kdc[8549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222943, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:09:03.562520 9284 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260924 04:09:03.563405 9284 webserver.cc:460] Webserver started at http://127.0.98.2:44703/ using document root <none> and password file <none>
I20260924 04:09:03.563974 9284 fs_manager.cc:359] Metadata directory not provided
I20260924 04:09:03.564080 9284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:09:03.567942 9284 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:09:03.570144 9299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:09:03.571511 9284 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:09:03.571635 9284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/ts-1/wal
uuid: "619e4d856d2446f39d05822025ec3db6"
format_stamp: "Formatted at 2026-09-24 04:09:02 on dist-test-slave-00ld"
server_key: "48ae6b102ef490043e165b5b074117b7"
server_key_iv: "316d863b23026a6c15ddddaeaac5f011"
server_key_version: "encryptionkey@0"
I20260924 04:09:03.572214 9284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:09:03.594727 9284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:09:03.597743 9284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:09:03.597949 9284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:09:03.598539 9284 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:09:03.599391 9284 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:09:03.599455 9284 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:09:03.599529 9284 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:09:03.599572 9284 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:09:03.608256 9284 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33257
I20260924 04:09:03.608326 9412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33257 every 8 connection(s)
I20260924 04:09:03.609385 9284 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 04:09:03 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:09:03 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:09:03.619589 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 9284
I20260924 04:09:03.620122 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:32865
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:42249
--webserver_port=43837
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:09:03.620550 9282 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:09:03.611180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46629 (local address 127.0.98.62:42249)
0924 04:09:03.611310 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 04:09:03.611313 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:09:03.612126 (+ 813us) server_negotiation.cc:407] Connection header received
0924 04:09:03.613607 (+ 1481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:09:03.613611 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:09:03.613746 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:09:03.613895 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:09:03.615055 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:03.615578 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:03.616736 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:03.616956 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:03.617064 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:03.620409 (+ 3345us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:46629: BlockingRecv error: recv got EOF from 127.0.98.2:46629 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 04:09:03.621305 9415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:09:03.611492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42249 (local address 127.0.98.2:46629)
0924 04:09:03.611951 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0924 04:09:03.611995 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 04:09:03.613312 (+ 1317us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:09:03.614083 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:09:03.614090 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:09:03.614434 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:09:03.614906 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:03.614919 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:03.615695 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:03.615699 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:03.616586 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:03.616595 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:03.617111 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:03.617115 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:03.617272 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:03.617806 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:09:03.617821 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:09:03.620240 (+ 2419us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42249: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":106,"threads_started":1}
W20260924 04:09:03.621863 9413 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42249 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:42249: Client connection negotiation failed: client connection to 127.0.98.62:42249: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 24 04:09:03 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:09:03 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:09:03.627985 9415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:09:03.622301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42249 (local address 127.0.98.2:60039)
0924 04:09:03.622402 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0924 04:09:03.622417 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:09:03.622492 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:09:03.622748 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:09:03.622750 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:09:03.622888 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:09:03.623020 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:03.623030 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:03.623702 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:03.623709 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:03.624758 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:03.624772 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:03.625212 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:03.625216 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:03.625348 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:03.625811 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:09:03.625831 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:09:03.627816 (+ 1985us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42249: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":29}
W20260924 04:09:03.628006 9282 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:09:03.622373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60039 (local address 127.0.98.62:42249)
0924 04:09:03.622528 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 04:09:03.622535 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 04:09:03.622552 (+ 17us) server_negotiation.cc:407] Connection header received
0924 04:09:03.622598 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:09:03.622601 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:09:03.622647 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:09:03.622734 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:09:03.623128 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:03.623596 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:03.624879 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:03.625097 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:03.625186 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:03.627918 (+ 2732us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:60039: BlockingRecv error: recv got EOF from 127.0.98.2:60039 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 24 04:09:03 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:09:03 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:09:03.633841 9415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:09:03.628492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42249 (local address 127.0.98.2:59901)
0924 04:09:03.628625 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 04:09:03.628643 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 04:09:03.628795 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:09:03.629025 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:09:03.629031 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:09:03.629202 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:09:03.629315 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:03.629324 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:03.629976 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:03.629983 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:03.630946 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:03.630958 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:03.631375 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:03.631381 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:03.631492 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:03.631908 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:09:03.631922 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:09:03.633728 (+ 1806us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42249: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":49}
W20260924 04:09:03.633913 9282 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:09:03.628607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59901 (local address 127.0.98.62:42249)
0924 04:09:03.628728 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0924 04:09:03.628731 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:09:03.628743 (+ 12us) server_negotiation.cc:407] Connection header received
0924 04:09:03.628886 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:09:03.628888 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:09:03.628921 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:09:03.628985 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:09:03.629420 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:03.629876 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:03.631077 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:03.631270 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:03.631342 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:03.633833 (+ 2491us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:59901: BlockingRecv error: recv got EOF from 127.0.98.2:59901 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260924 04:09:03.634248 9413 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 04:09:03.734117 9416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:09:03.734406 9416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:09:03.734491 9416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:09:03.738760 9416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:09:03.738847 9416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:09:03.738932 9416 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:09:03.744419 9416 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:32865
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=43837
--tserver_master_addrs=127.0.98.62:42249
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9416
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:09:03.745564 9416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:09:03.746605 9416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:09:03.756232 9421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:09:03.756349 9416 server_base.cc:1034] running on GCE node
W20260924 04:09:03.756258 9424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:09:03.756258 9422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:09:03.756999 9416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:09:03.757625 9416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:09:03.758813 9416 hybrid_clock.cc:648] HybridClock initialized: now 1790222943758799 us; error 37 us; skew 500 ppm
Sep 24 04:09:03 dist-test-slave-00ld krb5kdc[8549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222943, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:09:03.761619 9416 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260924 04:09:03.762511 9416 webserver.cc:460] Webserver started at http://127.0.98.3:43837/ using document root <none> and password file <none>
I20260924 04:09:03.763087 9416 fs_manager.cc:359] Metadata directory not provided
I20260924 04:09:03.763203 9416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:09:03.767216 9416 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260924 04:09:03.769389 9431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:09:03.770658 9416 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:09:03.770784 9416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/ts-2/wal
uuid: "0d6708cb99a94cc7ba024d14292d4e85"
format_stamp: "Formatted at 2026-09-24 04:09:02 on dist-test-slave-00ld"
server_key: "2fb5212dcf9c7ca4463b6d8cfd34d098"
server_key_iv: "40cd90bb63125b061b3f639d836add25"
server_key_version: "encryptionkey@0"
I20260924 04:09:03.771327 9416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:09:03.782424 9416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:09:03.785743 9416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:09:03.785928 9416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:09:03.786522 9416 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:09:03.787374 9416 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:09:03.787439 9416 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:09:03.787505 9416 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:09:03.787551 9416 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:09:03.795976 9416 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:32865
I20260924 04:09:03.796032 9544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:32865 every 8 connection(s)
I20260924 04:09:03.797056 9416 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790222898807720-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 04:09:03 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:09:03 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:09:03.807212 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 9416
W20260924 04:09:03.807447 9282 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:09:03.798818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34355 (local address 127.0.98.62:42249)
0924 04:09:03.798952 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 04:09:03.798955 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:09:03.799775 (+ 820us) server_negotiation.cc:407] Connection header received
0924 04:09:03.801125 (+ 1350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:09:03.801128 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:09:03.801176 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:09:03.801251 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:09:03.802357 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:03.802876 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:03.803928 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:03.804130 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:03.804204 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:03.807325 (+ 3121us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:34355: BlockingRecv error: recv got EOF from 127.0.98.3:34355 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 04:09:03.808144 9547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:09:03.799204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42249 (local address 127.0.98.3:34355)
0924 04:09:03.799616 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0924 04:09:03.799656 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 04:09:03.800924 (+ 1268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:09:03.801410 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:09:03.801417 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:09:03.801767 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:09:03.802237 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:03.802249 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:03.802975 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:03.802978 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:03.803827 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:03.803835 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:03.804247 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:03.804249 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:03.804383 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:03.804973 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:09:03.804989 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:09:03.807209 (+ 2220us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42249: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":108,"threads_started":1}
W20260924 04:09:03.808732 9545 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42249 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:42249: Client connection negotiation failed: client connection to 127.0.98.62:42249: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 24 04:09:03 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:09:03 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:09:03.814870 9547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:09:03.809198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42249 (local address 127.0.98.3:58739)
0924 04:09:03.809295 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0924 04:09:03.809310 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:09:03.809387 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:09:03.809706 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:09:03.809708 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:09:03.809867 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:09:03.809967 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:03.809973 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:03.810861 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:03.810864 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:03.811528 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:03.811537 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:03.812234 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:03.812238 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:03.812359 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:03.812795 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:09:03.812806 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:09:03.814777 (+ 1971us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42249: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":30}
W20260924 04:09:03.815294 9549 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:09:03.809249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58739 (local address 127.0.98.62:42249)
0924 04:09:03.809505 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0924 04:09:03.809510 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:09:03.809524 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:09:03.809564 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:09:03.809566 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:09:03.809598 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:09:03.809666 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:09:03.810073 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:03.810752 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:03.811673 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:03.812123 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:03.812204 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:03.814916 (+ 2712us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:58739: BlockingRecv error: recv got EOF from 127.0.98.3:58739 (error 108)
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":171,"thread_start_us":84,"threads_started":1}
I20260924 04:09:03.817763 9282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:09:03.808238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39100 (local address 127.0.98.62:42249)
0924 04:09:03.808351 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0924 04:09:03.808355 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:09:03.808368 (+ 13us) server_negotiation.cc:407] Connection header received
0924 04:09:03.808409 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:09:03.808412 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:09:03.808473 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:09:03.808552 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:09:03.809530 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:03.810261 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:03.811269 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:03.811599 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:03.811718 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:03.813313 (+ 1595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:09:03.813323 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:09:03.813349 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:09:03.813380 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:09:03.815297 (+ 1917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:09:03.816028 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:09:03.816035 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:09:03.816042 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:09:03.816195 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:09:03.816477 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:09:03.816480 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:09:03.816481 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:09:03.816891 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:09:03.817039 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 04:09:03.817356 (+ 317us) server_negotiation.cc:299] Negotiation successful
0924 04:09:03.817604 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":48}
Sep 24 04:09:03 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:09:03 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:09:03.821146 9547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:09:03.815427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42249 (local address 127.0.98.3:44899)
0924 04:09:03.815553 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 04:09:03.815564 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:09:03.815644 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:09:03.815889 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:09:03.815892 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:09:03.816042 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:09:03.816129 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:03.816134 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:03.817243 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:03.817246 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:03.818007 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:03.818016 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:03.818491 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:03.818494 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:03.818547 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:03.818991 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:09:03.819004 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:09:03.821009 (+ 2005us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42249: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":46}
W20260924 04:09:03.821209 9549 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:09:03.815547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44899 (local address 127.0.98.62:42249)
0924 04:09:03.815716 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 04:09:03.815720 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:09:03.815731 (+ 11us) server_negotiation.cc:407] Connection header received
0924 04:09:03.815760 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:09:03.815768 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:09:03.815800 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:09:03.815877 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:09:03.816265 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:03.817119 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:03.818104 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:03.818395 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:03.818555 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:03.821109 (+ 2554us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:44899: BlockingRecv error: recv got EOF from 127.0.98.3:44899 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
W20260924 04:09:03.821368 9545 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 24 04:09:04 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:09:04 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:09:04.447750 9550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:09:04.441919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42249 (local address 127.0.98.1:45971)
0924 04:09:04.442187 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0924 04:09:04.442201 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:09:04.442300 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:09:04.442579 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:09:04.442581 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:09:04.442757 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:09:04.442874 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:04.442880 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:04.443781 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:04.443784 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:04.444367 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:04.444375 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:04.444837 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:04.444840 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:04.444920 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:04.445433 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:09:04.445444 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:09:04.447616 (+ 2172us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42249: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":87,"threads_started":1}
W20260924 04:09:04.447880 9551 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:09:04.441987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45971 (local address 127.0.98.62:42249)
0924 04:09:04.442214 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0924 04:09:04.442217 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:09:04.442254 (+ 37us) server_negotiation.cc:407] Connection header received
0924 04:09:04.442386 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:09:04.442389 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:09:04.442431 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:09:04.442513 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:09:04.443061 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:04.443659 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:04.444467 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:04.444736 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:04.444836 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:04.447753 (+ 2917us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:45971: BlockingRecv error: recv got EOF from 127.0.98.1:45971 (error 108)
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":84,"threads_started":1}
Sep 24 04:09:04 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:09:04 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:09:04.639848 9552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:09:04.634795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42249 (local address 127.0.98.2:55051)
0924 04:09:04.635052 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0924 04:09:04.635065 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:09:04.635163 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:09:04.635386 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:09:04.635388 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:09:04.635563 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:09:04.635655 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:04.635659 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:04.636336 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:04.636338 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:04.636943 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:04.636951 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:04.637341 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:04.637343 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:04.637402 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:04.637871 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:09:04.637882 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:09:04.639720 (+ 1838us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42249: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":66,"threads_started":1}
W20260924 04:09:04.640009 9551 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:09:04.634875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55051 (local address 127.0.98.62:42249)
0924 04:09:04.635061 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 04:09:04.635065 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:09:04.635122 (+ 57us) server_negotiation.cc:407] Connection header received
0924 04:09:04.635247 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:09:04.635249 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:09:04.635291 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:09:04.635355 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:09:04.635743 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:04.636251 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:04.637035 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:04.637251 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:04.637326 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:04.639856 (+ 2530us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:55051: BlockingRecv error: recv got EOF from 127.0.98.2:55051 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 24 04:09:04 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:09:04 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:09:04.827471 9553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:09:04.821967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42249 (local address 127.0.98.3:43565)
0924 04:09:04.822269 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0924 04:09:04.822286 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 04:09:04.822398 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:09:04.822624 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:09:04.822626 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:09:04.822816 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:09:04.822955 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:04.822962 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:04.823657 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:04.823660 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:04.824309 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:04.824317 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:04.824734 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:04.824736 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:04.824829 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:04.825312 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:09:04.825324 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:09:04.827335 (+ 2011us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42249: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":94,"threads_started":1}
W20260924 04:09:04.827594 9551 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:09:04.822053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43565 (local address 127.0.98.62:42249)
0924 04:09:04.822185 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 04:09:04.822194 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 04:09:04.822362 (+ 168us) server_negotiation.cc:407] Connection header received
0924 04:09:04.822476 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:09:04.822478 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:09:04.822519 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:09:04.822583 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:09:04.823053 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:04.823562 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:04.824409 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:04.824612 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:04.824735 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:04.827461 (+ 2726us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:43565: BlockingRecv error: recv got EOF from 127.0.98.3:43565 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 24 04:09:05 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:09:05 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:09:05.453718 9554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:09:05.448537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42249 (local address 127.0.98.1:36759)
0924 04:09:05.448870 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0924 04:09:05.448883 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:09:05.448976 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:09:05.449219 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:09:05.449221 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:09:05.449391 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:09:05.449526 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:05.449532 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:05.450231 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:05.450234 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:05.450830 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:05.450838 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:05.451228 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:05.451230 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:05.451290 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:05.451755 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:09:05.451766 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:09:05.453589 (+ 1823us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42249: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":81,"threads_started":1}
W20260924 04:09:05.453840 9555 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:09:05.448604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36759 (local address 127.0.98.62:42249)
0924 04:09:05.448902 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0924 04:09:05.448906 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:09:05.448923 (+ 17us) server_negotiation.cc:407] Connection header received
0924 04:09:05.449069 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:09:05.449071 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:09:05.449113 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:09:05.449188 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:09:05.449619 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:05.450137 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:05.450925 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:05.451137 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:05.451207 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:05.453708 (+ 2501us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:36759: BlockingRecv error: recv got EOF from 127.0.98.1:36759 (error 108)
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":130,"threads_started":1}
Sep 24 04:09:05 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:09:05 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:09:05.646253 9556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:09:05.640811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42249 (local address 127.0.98.2:42787)
0924 04:09:05.641065 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0924 04:09:05.641086 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 04:09:05.641175 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:09:05.641427 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:09:05.641430 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:09:05.641608 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:09:05.641703 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:05.641708 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:05.642448 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:05.642451 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:05.643048 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:05.643057 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:05.643542 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:05.643546 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:05.643631 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:05.644213 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:09:05.644225 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:09:05.646060 (+ 1835us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42249: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":71,"threads_started":1}
W20260924 04:09:05.646291 9555 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:09:05.640958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42787 (local address 127.0.98.62:42249)
0924 04:09:05.641115 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 04:09:05.641118 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:09:05.641149 (+ 31us) server_negotiation.cc:407] Connection header received
0924 04:09:05.641267 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:09:05.641269 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:09:05.641314 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:09:05.641380 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:09:05.641800 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:05.642309 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:05.643194 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:05.643427 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:05.643539 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:05.646185 (+ 2646us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:42787: BlockingRecv error: recv got EOF from 127.0.98.2:42787 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 24 04:09:05 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:09:05 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:09:05.834074 9557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:09:05.828327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42249 (local address 127.0.98.3:39761)
0924 04:09:05.828754 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0924 04:09:05.828769 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:09:05.828876 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:09:05.829152 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:09:05.829155 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:09:05.829378 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:09:05.829516 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:05.829526 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:05.830406 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:05.830409 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:05.830950 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:05.830957 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:05.831401 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:05.831404 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:05.831462 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:05.832000 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:09:05.832011 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:09:05.833873 (+ 1862us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42249: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":118,"threads_started":1}
W20260924 04:09:05.834232 9555 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:09:05.828549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39761 (local address 127.0.98.62:42249)
0924 04:09:05.828744 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 04:09:05.828748 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:09:05.828843 (+ 95us) server_negotiation.cc:407] Connection header received
0924 04:09:05.828958 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:09:05.828961 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:09:05.829031 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:09:05.829121 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:09:05.829639 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:05.830311 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:05.831076 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:05.831304 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:05.831399 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:05.834067 (+ 2668us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:39761: BlockingRecv error: recv got EOF from 127.0.98.3:39761 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 24 04:09:06 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:09:06 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:09:06.459622 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:09:06.454509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42249 (local address 127.0.98.1:56299)
0924 04:09:06.454775 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0924 04:09:06.454791 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:09:06.454881 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:09:06.455120 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:09:06.455122 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:09:06.455292 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:09:06.455386 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:06.455392 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:06.456102 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:06.456105 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:06.456713 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:06.456721 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:06.457121 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:06.457123 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:06.457182 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:06.457685 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:09:06.457696 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:09:06.459505 (+ 1809us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42249: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":105,"threads_started":1}
W20260924 04:09:06.459805 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:09:06.454593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56299 (local address 127.0.98.62:42249)
0924 04:09:06.454819 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0924 04:09:06.454822 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:09:06.454839 (+ 17us) server_negotiation.cc:407] Connection header received
0924 04:09:06.454963 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:09:06.454965 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:09:06.455007 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:09:06.455093 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:09:06.455484 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:06.456009 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:06.456806 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:06.457030 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:06.457106 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:06.459635 (+ 2529us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:56299: BlockingRecv error: recv got EOF from 127.0.98.1:56299 (error 108)
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":88,"threads_started":1}
Sep 24 04:09:06 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:09:06 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:09:06.653038 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:09:06.647182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42249 (local address 127.0.98.2:39587)
0924 04:09:06.647456 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0924 04:09:06.647470 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:09:06.647573 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:09:06.647930 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:09:06.647933 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:09:06.648139 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:09:06.648232 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:06.648237 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:06.649047 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:06.649051 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:06.649661 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:06.649669 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:06.650113 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:06.650116 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:06.650199 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:06.650824 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:09:06.650835 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:09:06.652902 (+ 2067us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42249: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":94,"threads_started":1}
W20260924 04:09:06.653173 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:09:06.647347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39587 (local address 127.0.98.62:42249)
0924 04:09:06.647483 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 04:09:06.647486 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:09:06.647525 (+ 39us) server_negotiation.cc:407] Connection header received
0924 04:09:06.647691 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:09:06.647694 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:09:06.647743 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:09:06.647871 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:09:06.648347 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:06.648936 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:06.649776 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:06.650005 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:06.650108 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:06.653047 (+ 2939us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:39587: BlockingRecv error: recv got EOF from 127.0.98.2:39587 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 24 04:09:06 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:09:06 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:09:06.840219 9561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:09:06.834817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42249 (local address 127.0.98.3:48059)
0924 04:09:06.835090 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0924 04:09:06.835112 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 04:09:06.835216 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:09:06.835473 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:09:06.835476 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:09:06.835651 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:09:06.835745 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:06.835750 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:06.836511 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:06.836514 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:06.837123 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:06.837132 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:06.837650 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:06.837658 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:06.837726 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:06.838280 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:09:06.838292 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:09:06.840106 (+ 1814us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42249: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":112,"threads_started":1}
W20260924 04:09:06.840353 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:09:06.834890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48059 (local address 127.0.98.62:42249)
0924 04:09:06.835016 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 04:09:06.835019 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:09:06.835186 (+ 167us) server_negotiation.cc:407] Connection header received
0924 04:09:06.835293 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:09:06.835295 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:09:06.835342 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:09:06.835415 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:09:06.835845 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:06.836378 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:06.837221 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:06.837525 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:06.837605 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:06.840230 (+ 2625us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:48059: BlockingRecv error: recv got EOF from 127.0.98.3:48059 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 24 04:09:07 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:09:07 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:09:07.466137 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:09:07.460406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42249 (local address 127.0.98.1:34377)
0924 04:09:07.460701 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0924 04:09:07.460720 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 04:09:07.460828 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:09:07.461051 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:09:07.461053 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:09:07.461226 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:09:07.461319 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:07.461324 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:07.462045 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:07.462048 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:07.462598 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:07.462605 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:07.463088 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:07.463091 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:07.463151 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:07.463765 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:09:07.463784 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:09:07.465957 (+ 2173us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42249: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":68,"threads_started":1}
W20260924 04:09:07.466199 9563 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:09:07.460550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34377 (local address 127.0.98.62:42249)
0924 04:09:07.460842 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0924 04:09:07.460845 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:09:07.460859 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:09:07.460896 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:09:07.460898 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:09:07.460938 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:09:07.461015 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:09:07.461421 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:07.461953 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:07.462718 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:07.462965 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:07.463053 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:07.466075 (+ 3022us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:34377: BlockingRecv error: recv got EOF from 127.0.98.1:34377 (error 108)
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":121,"threads_started":1}
Sep 24 04:09:07 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:09:07 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:09:07.659406 9564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:09:07.653838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42249 (local address 127.0.98.2:40191)
0924 04:09:07.654102 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0924 04:09:07.654116 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:09:07.654217 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:09:07.654478 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:09:07.654480 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:09:07.654655 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:09:07.654746 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:07.654751 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:07.655475 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:07.655477 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:07.656047 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:07.656054 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:07.656517 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:07.656520 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:07.656580 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:07.657121 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:09:07.657132 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:09:07.659196 (+ 2064us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42249: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":103,"threads_started":1}
W20260924 04:09:07.659428 9563 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:09:07.653882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40191 (local address 127.0.98.62:42249)
0924 04:09:07.654027 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 04:09:07.654031 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:09:07.654186 (+ 155us) server_negotiation.cc:407] Connection header received
0924 04:09:07.654328 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:09:07.654330 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:09:07.654374 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:09:07.654443 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:09:07.654838 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:07.655385 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:07.656154 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:07.656419 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:07.656501 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:07.659329 (+ 2828us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:40191: BlockingRecv error: recv got EOF from 127.0.98.2:40191 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 24 04:09:07 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:09:07 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:09:07.846338 9565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:09:07.841052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42249 (local address 127.0.98.3:32773)
0924 04:09:07.841315 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0924 04:09:07.841333 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 04:09:07.841442 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:09:07.841678 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:09:07.841680 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:09:07.841877 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:09:07.841973 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:07.841978 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:07.842710 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:07.842731 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:07.843301 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:07.843311 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:07.843696 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:07.843698 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:07.843770 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:07.844352 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:09:07.844364 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:09:07.846162 (+ 1798us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42249: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":104,"threads_started":1}
W20260924 04:09:07.846410 9563 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:09:07.841159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:32773 (local address 127.0.98.62:42249)
0924 04:09:07.841291 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 04:09:07.841302 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 04:09:07.841414 (+ 112us) server_negotiation.cc:407] Connection header received
0924 04:09:07.841525 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:09:07.841527 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:09:07.841571 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:09:07.841642 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:09:07.842070 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:07.842606 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:07.843400 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:07.843603 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:07.843673 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:07.846297 (+ 2624us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:32773: BlockingRecv error: recv got EOF from 127.0.98.3:32773 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 24 04:09:08 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:09:08 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:09:08.476900 9566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:09:08.466913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42249 (local address 127.0.98.1:49745)
0924 04:09:08.467211 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0924 04:09:08.467225 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:09:08.467313 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:09:08.467558 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:09:08.467560 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:09:08.467748 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:09:08.467845 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:08.467850 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:08.468767 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:08.468769 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:08.469369 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:08.469378 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:08.469921 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:08.469924 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:08.469993 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:08.474488 (+ 4495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:09:08.474500 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:09:08.476762 (+ 2262us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42249: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":86,"threads_started":1}
W20260924 04:09:08.477006 9567 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:09:08.467018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49745 (local address 127.0.98.62:42249)
0924 04:09:08.467278 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0924 04:09:08.467281 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:09:08.467294 (+ 13us) server_negotiation.cc:407] Connection header received
0924 04:09:08.467404 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:09:08.467406 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:09:08.467447 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:09:08.467534 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:09:08.467949 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:08.468661 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:08.469478 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:08.469771 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:08.469851 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:08.476880 (+ 7029us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:49745: BlockingRecv error: recv got EOF from 127.0.98.1:49745 (error 108)
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":84,"threads_started":1}
Sep 24 04:09:08 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:09:08 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:09:08.665495 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:09:08.660201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42249 (local address 127.0.98.2:41277)
0924 04:09:08.660459 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0924 04:09:08.660473 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:09:08.660555 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:09:08.660838 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:09:08.660840 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:09:08.661021 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:09:08.661111 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:08.661116 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:08.661871 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:08.661874 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:08.662460 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:08.662467 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:08.662936 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:08.662939 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:08.662999 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:08.663485 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:09:08.663496 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:09:08.665382 (+ 1886us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42249: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":73,"threads_started":1}
W20260924 04:09:08.665697 9567 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:09:08.660343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41277 (local address 127.0.98.62:42249)
0924 04:09:08.660502 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 04:09:08.660505 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:09:08.660518 (+ 13us) server_negotiation.cc:407] Connection header received
0924 04:09:08.660665 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:09:08.660667 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:09:08.660733 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:09:08.660800 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:09:08.661226 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:08.661755 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:08.662609 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:08.662834 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:08.662921 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:08.665528 (+ 2607us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:41277: BlockingRecv error: recv got EOF from 127.0.98.2:41277 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 24 04:09:08 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:09:08 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:09:08.852365 9569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:09:08.847291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42249 (local address 127.0.98.3:36135)
0924 04:09:08.847564 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0924 04:09:08.847577 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:09:08.847666 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:09:08.847896 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:09:08.847898 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:09:08.848079 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:09:08.848173 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:08.848178 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:08.848902 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:08.848904 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:08.849460 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:08.849467 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:08.849893 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:08.849896 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:08.849957 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:08.850459 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:09:08.850470 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:09:08.852240 (+ 1770us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42249: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":110,"threads_started":1}
W20260924 04:09:08.852545 9567 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:09:08.847391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36135 (local address 127.0.98.62:42249)
0924 04:09:08.847533 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 04:09:08.847536 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:09:08.847628 (+ 92us) server_negotiation.cc:407] Connection header received
0924 04:09:08.847730 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:09:08.847737 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:09:08.847780 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:09:08.847857 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:09:08.848273 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:08.848808 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:08.849558 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:08.849783 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:08.849859 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:08.852375 (+ 2516us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:36135: BlockingRecv error: recv got EOF from 127.0.98.3:36135 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 24 04:09:09 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:09:09 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:09:09.483548 9570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:09:09.477743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42249 (local address 127.0.98.1:53137)
0924 04:09:09.478054 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0924 04:09:09.478072 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 04:09:09.478184 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:09:09.478424 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:09:09.478426 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:09:09.478599 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:09:09.478692 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:09.478696 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:09.479414 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:09.479417 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:09.479988 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:09.479995 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:09.480447 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:09.480450 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:09.480509 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:09.481157 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:09:09.481168 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:09:09.483367 (+ 2199us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42249: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":80,"threads_started":1}
W20260924 04:09:09.483713 9571 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:09:09.477815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53137 (local address 127.0.98.62:42249)
0924 04:09:09.478080 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0924 04:09:09.478083 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:09:09.478156 (+ 73us) server_negotiation.cc:407] Connection header received
0924 04:09:09.478263 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:09:09.478265 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:09:09.478307 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:09:09.478386 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:09:09.478785 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:09.479327 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:09.480093 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:09.480345 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:09.480448 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:09.483588 (+ 3140us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:53137: BlockingRecv error: recv got EOF from 127.0.98.1:53137 (error 108)
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":94,"threads_started":1}
Sep 24 04:09:09 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:09:09 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:09:09.676331 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:09:09.666231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42249 (local address 127.0.98.2:58763)
0924 04:09:09.666539 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0924 04:09:09.666555 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:09:09.666693 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:09:09.666976 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:09:09.666979 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:09:09.667189 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:09:09.667281 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:09.667286 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:09.668056 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:09.668062 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:09.668744 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:09.668753 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:09.669259 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:09.669262 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:09.669358 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:09.674039 (+ 4681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:09:09.674051 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:09:09.676212 (+ 2161us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42249: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":85,"threads_started":1}
W20260924 04:09:09.676507 9571 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:09:09.666364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58763 (local address 127.0.98.62:42249)
0924 04:09:09.666517 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0924 04:09:09.666522 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:09:09.666635 (+ 113us) server_negotiation.cc:407] Connection header received
0924 04:09:09.666825 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:09:09.666828 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:09:09.666872 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:09:09.666954 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:09:09.667420 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:09.667948 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:09.668861 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:09.669146 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:09.669258 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:09.676341 (+ 7083us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:58763: BlockingRecv error: recv got EOF from 127.0.98.2:58763 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 24 04:09:09 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:09:09 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:09:09.858610 9573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:09:09.853340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42249 (local address 127.0.98.3:51833)
0924 04:09:09.853599 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0924 04:09:09.853615 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:09:09.853714 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:09:09.853965 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:09:09.853968 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:09:09.854145 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:09:09.854241 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:09.854246 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:09.855126 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:09.855129 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:09.855686 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:09.855694 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:09.856115 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:09.856117 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:09.856179 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:09.856707 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:09:09.856719 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:09:09.858492 (+ 1773us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42249: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":96,"threads_started":1}
W20260924 04:09:09.858805 9571 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:09:09.853450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51833 (local address 127.0.98.62:42249)
0924 04:09:09.853587 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 04:09:09.853591 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:09:09.853667 (+ 76us) server_negotiation.cc:407] Connection header received
0924 04:09:09.853785 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:09:09.853787 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:09:09.853829 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:09:09.853900 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:09:09.854354 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:09.855026 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:09.855796 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:09.856009 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:09.856100 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:09.858644 (+ 2544us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:51833: BlockingRecv error: recv got EOF from 127.0.98.3:51833 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 24 04:09:10 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:09:10 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:09:10.489879 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:09:10.484340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42249 (local address 127.0.98.1:54701)
0924 04:09:10.484720 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0924 04:09:10.484744 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 04:09:10.484867 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:09:10.485245 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:09:10.485248 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:09:10.485432 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:09:10.485538 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:10.485543 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:10.486283 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:10.486286 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:10.486862 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:10.486870 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:10.487367 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:10.487370 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:10.487429 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:10.487920 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:09:10.487931 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:09:10.489765 (+ 1834us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42249: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":81,"threads_started":1}
W20260924 04:09:10.490041 9575 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:09:10.484472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54701 (local address 127.0.98.62:42249)
0924 04:09:10.484809 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0924 04:09:10.484812 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:09:10.484826 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:09:10.484953 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:09:10.484956 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:09:10.484999 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:09:10.485096 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:09:10.485649 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:10.486185 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:10.486974 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:10.487259 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:10.487348 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:10.489902 (+ 2554us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:54701: BlockingRecv error: recv got EOF from 127.0.98.1:54701 (error 108)
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":91,"threads_started":1}
Sep 24 04:09:10 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:09:10 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:09:10.682852 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:09:10.677151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42249 (local address 127.0.98.2:36119)
0924 04:09:10.677449 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0924 04:09:10.677462 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:09:10.677627 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:09:10.677905 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:09:10.677907 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:09:10.678175 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:09:10.678276 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:10.678282 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:10.679052 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:10.679057 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:10.679615 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:10.679623 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:10.680089 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:10.680093 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:10.680177 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:10.680667 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:09:10.680709 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:09:10.682722 (+ 2013us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42249: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":72,"threads_started":1}
W20260924 04:09:10.683015 9575 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:09:10.677248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36119 (local address 127.0.98.62:42249)
0924 04:09:10.677439 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 04:09:10.677443 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:09:10.677569 (+ 126us) server_negotiation.cc:407] Connection header received
0924 04:09:10.677735 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:09:10.677739 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:09:10.677793 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:09:10.677883 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:09:10.678402 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:10.678932 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:10.679745 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:10.679974 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:10.680085 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:10.682855 (+ 2770us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:36119: BlockingRecv error: recv got EOF from 127.0.98.2:36119 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 24 04:09:10 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:09:10 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:09:10.865208 9577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:09:10.859447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42249 (local address 127.0.98.3:41507)
0924 04:09:10.859790 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0924 04:09:10.859804 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:09:10.859901 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:09:10.860149 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:09:10.860151 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:09:10.860328 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:09:10.860426 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:10.860432 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:10.861169 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:10.861172 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:10.861792 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:10.861801 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:10.862219 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:10.862221 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:10.862284 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:10.862767 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:09:10.862779 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:09:10.865055 (+ 2276us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42249: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":123,"threads_started":1}
W20260924 04:09:10.865250 9575 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:09:10.859510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41507 (local address 127.0.98.62:42249)
0924 04:09:10.859633 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 04:09:10.859637 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:09:10.859870 (+ 233us) server_negotiation.cc:407] Connection header received
0924 04:09:10.859965 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:09:10.859968 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:09:10.860011 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:09:10.860078 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:09:10.860527 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:10.861054 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:10.861913 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:10.862121 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:10.862194 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:10.865161 (+ 2967us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:41507: BlockingRecv error: recv got EOF from 127.0.98.3:41507 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 24 04:09:11 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:09:11 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:09:11.496171 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:09:11.490628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42249 (local address 127.0.98.1:55455)
0924 04:09:11.490885 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0924 04:09:11.490899 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:09:11.490984 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:09:11.491274 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:09:11.491276 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:09:11.491448 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:09:11.491541 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:11.491546 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:11.492286 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:11.492288 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:11.492918 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:11.492926 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:11.493549 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:11.493552 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:11.493649 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:11.494272 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:09:11.494284 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:09:11.496057 (+ 1773us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42249: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":94,"threads_started":1}
W20260924 04:09:11.496390 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:09:11.490840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55455 (local address 127.0.98.62:42249)
0924 04:09:11.491067 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0924 04:09:11.491071 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:09:11.491082 (+ 11us) server_negotiation.cc:407] Connection header received
0924 04:09:11.491118 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:09:11.491120 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:09:11.491159 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:09:11.491242 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:09:11.491641 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:11.492196 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:11.493103 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:11.493414 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:11.493518 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:11.496225 (+ 2707us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:55455: BlockingRecv error: recv got EOF from 127.0.98.1:55455 (error 108)
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":82,"threads_started":1}
Sep 24 04:09:11 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:09:11 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:09:11.689302 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:09:11.683658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42249 (local address 127.0.98.2:44233)
0924 04:09:11.683941 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0924 04:09:11.683954 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:09:11.684054 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:09:11.684264 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:09:11.684266 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:09:11.684436 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:09:11.684525 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:11.684530 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:11.685457 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:11.685459 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:11.686014 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:11.686022 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:11.686527 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:11.686530 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:11.686590 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:11.687146 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:09:11.687157 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:09:11.689126 (+ 1969us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42249: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":90,"threads_started":1}
W20260924 04:09:11.689350 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:09:11.683724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44233 (local address 127.0.98.62:42249)
0924 04:09:11.683869 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 04:09:11.683876 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 04:09:11.684031 (+ 155us) server_negotiation.cc:407] Connection header received
0924 04:09:11.684119 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:09:11.684122 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:09:11.684164 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:09:11.684231 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:09:11.684621 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:11.685340 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:11.686136 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:11.686361 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:11.686540 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:11.689248 (+ 2708us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:44233: BlockingRecv error: recv got EOF from 127.0.98.2:44233 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 24 04:09:11 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:09:11 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:09:11.871555 9581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:09:11.866003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42249 (local address 127.0.98.3:51045)
0924 04:09:11.866319 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0924 04:09:11.866341 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 04:09:11.866466 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:09:11.866747 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:09:11.866749 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:09:11.866954 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:09:11.867038 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:11.867044 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:11.867797 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:11.867802 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:11.868397 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:11.868405 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:11.869012 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:11.869015 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:11.869100 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:11.869568 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:09:11.869581 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:09:11.871385 (+ 1804us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42249: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":84,"threads_started":1}
W20260924 04:09:11.871613 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:09:11.866116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51045 (local address 127.0.98.62:42249)
0924 04:09:11.866273 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 04:09:11.866278 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:09:11.866410 (+ 132us) server_negotiation.cc:407] Connection header received
0924 04:09:11.866572 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:09:11.866575 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:09:11.866624 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:09:11.866727 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:09:11.867153 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:11.867687 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:11.868523 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:11.868916 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:11.869007 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:11.871504 (+ 2497us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:51045: BlockingRecv error: recv got EOF from 127.0.98.3:51045 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 24 04:09:12 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:09:12 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:09:12.502696 9582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:09:12.496985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42249 (local address 127.0.98.1:45865)
0924 04:09:12.497283 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0924 04:09:12.497296 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:09:12.497384 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:09:12.497633 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:09:12.497635 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:09:12.497806 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:09:12.497904 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:12.497909 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:12.498803 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:12.498806 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:12.499505 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:12.499514 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:12.499937 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:12.499939 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:12.500039 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:12.500509 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:09:12.500521 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:09:12.502562 (+ 2041us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42249: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":87,"threads_started":1}
W20260924 04:09:12.502820 9583 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:09:12.497115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45865 (local address 127.0.98.62:42249)
0924 04:09:12.497334 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0924 04:09:12.497337 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:09:12.497350 (+ 13us) server_negotiation.cc:407] Connection header received
0924 04:09:12.497475 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:09:12.497477 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:09:12.497519 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:09:12.497598 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:09:12.498010 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:12.498687 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:12.499602 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:12.499831 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:12.499909 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:12.502695 (+ 2786us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:45865: BlockingRecv error: recv got EOF from 127.0.98.1:45865 (error 108)
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":56,"threads_started":1}
Sep 24 04:09:12 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:09:12 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:09:12.695642 9584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:09:12.690116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42249 (local address 127.0.98.2:35243)
0924 04:09:12.690381 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0924 04:09:12.690395 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:09:12.690503 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:09:12.690749 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:09:12.690752 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:09:12.690980 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:09:12.691081 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:12.691087 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:12.691809 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:12.691812 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:12.692371 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:12.692380 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:12.692937 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:12.692940 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:12.693009 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:12.693499 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:09:12.693510 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:09:12.695467 (+ 1957us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42249: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":92,"threads_started":1}
W20260924 04:09:12.695719 9583 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:09:12.690182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35243 (local address 127.0.98.62:42249)
0924 04:09:12.690329 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 04:09:12.690332 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:09:12.690463 (+ 131us) server_negotiation.cc:407] Connection header received
0924 04:09:12.690582 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:09:12.690584 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:09:12.690626 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:09:12.690700 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:09:12.691185 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:12.691693 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:12.692469 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:12.692671 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:12.692810 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:12.695610 (+ 2800us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:35243: BlockingRecv error: recv got EOF from 127.0.98.2:35243 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 24 04:09:12 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:09:12 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:09:12.877789 9585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:09:12.872347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42249 (local address 127.0.98.3:50223)
0924 04:09:12.872611 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0924 04:09:12.872625 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:09:12.872743 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:09:12.872979 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:09:12.872982 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:09:12.873167 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:09:12.873261 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:12.873266 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:12.874007 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:12.874012 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:12.874577 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:12.874585 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:12.875004 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:12.875008 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:12.875124 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:12.875634 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:09:12.875645 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:09:12.877610 (+ 1965us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42249: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":97,"threads_started":1}
W20260924 04:09:12.877826 9583 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:09:12.872427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50223 (local address 127.0.98.62:42249)
0924 04:09:12.872559 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 04:09:12.872563 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:09:12.872692 (+ 129us) server_negotiation.cc:407] Connection header received
0924 04:09:12.872819 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:09:12.872821 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:09:12.872864 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:09:12.872931 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:09:12.873355 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:12.873876 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:12.874700 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:12.874908 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:12.874979 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:12.877728 (+ 2749us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:50223: BlockingRecv error: recv got EOF from 127.0.98.3:50223 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 24 04:09:13 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:09:13 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:09:13.513175 9586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:09:13.503596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42249 (local address 127.0.98.1:38049)
0924 04:09:13.503913 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0924 04:09:13.503927 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:09:13.504014 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:09:13.504253 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:09:13.504255 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:09:13.504438 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:09:13.504529 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:13.504534 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:13.505287 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:13.505290 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:13.505843 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:13.505850 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:13.506275 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:13.506278 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:13.506337 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:13.511193 (+ 4856us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:09:13.511205 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:09:13.513020 (+ 1815us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42249: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":91,"threads_started":1}
W20260924 04:09:13.513378 9587 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:09:13.503748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38049 (local address 127.0.98.62:42249)
0924 04:09:13.503995 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0924 04:09:13.503999 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:09:13.504013 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:09:13.504100 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:09:13.504103 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:09:13.504144 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:09:13.504220 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:09:13.504629 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:13.505194 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:13.505939 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:13.506171 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:13.506243 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:13.513212 (+ 6969us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:38049: BlockingRecv error: recv got EOF from 127.0.98.1:38049 (error 108)
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":59,"threads_started":1}
Sep 24 04:09:13 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:09:13 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:09:13.701889 9588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:09:13.696449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42249 (local address 127.0.98.2:47643)
0924 04:09:13.696777 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0924 04:09:13.696792 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:09:13.696906 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:09:13.697183 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:09:13.697187 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:09:13.697405 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:09:13.697495 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:13.697500 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:13.698264 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:13.698269 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:13.698835 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:13.698842 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:13.699290 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:13.699294 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:13.699387 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:13.699948 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:09:13.699960 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:09:13.701776 (+ 1816us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42249: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":82,"threads_started":1}
W20260924 04:09:13.702207 9587 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:09:13.696476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47643 (local address 127.0.98.62:42249)
0924 04:09:13.696619 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 04:09:13.696625 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:09:13.696857 (+ 232us) server_negotiation.cc:407] Connection header received
0924 04:09:13.697013 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:09:13.697017 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:09:13.697070 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:09:13.697157 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:09:13.697613 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:13.698154 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:13.698955 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:13.699179 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:13.699287 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:13.701987 (+ 2700us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:47643: BlockingRecv error: recv got EOF from 127.0.98.2:47643 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 24 04:09:13 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:09:13 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:09:13.884228 9589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:09:13.878627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42249 (local address 127.0.98.3:48625)
0924 04:09:13.878931 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0924 04:09:13.878966 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 04:09:13.879070 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:09:13.879342 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:09:13.879344 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:09:13.879531 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:09:13.879628 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:13.879633 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:13.880630 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:13.880632 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:13.881414 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:13.881423 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:13.881813 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:13.881815 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:13.881871 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:13.882421 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:09:13.882432 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:09:13.884117 (+ 1685us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42249: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":77,"threads_started":1}
W20260924 04:09:13.884385 9587 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:09:13.878796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48625 (local address 127.0.98.62:42249)
0924 04:09:13.878934 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 04:09:13.878938 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:09:13.879034 (+ 96us) server_negotiation.cc:407] Connection header received
0924 04:09:13.879148 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:09:13.879150 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:09:13.879204 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:09:13.879273 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:09:13.879766 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:13.880489 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:13.881509 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:13.881724 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:13.881802 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:13.884236 (+ 2434us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:48625: BlockingRecv error: recv got EOF from 127.0.98.3:48625 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 24 04:09:14 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:09:14 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:09:14.518975 9590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:09:14.513938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42249 (local address 127.0.98.1:44495)
0924 04:09:14.514201 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0924 04:09:14.514214 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:09:14.514300 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:09:14.514548 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:09:14.514550 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:09:14.514725 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:09:14.514819 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:14.514823 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:14.515542 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:14.515544 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:14.516095 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:14.516102 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:14.516524 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:14.516527 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:14.516586 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:14.517148 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:09:14.517159 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:09:14.518865 (+ 1706us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42249: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":91,"threads_started":1}
W20260924 04:09:14.519140 9591 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:09:14.514021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44495 (local address 127.0.98.62:42249)
0924 04:09:14.514285 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0924 04:09:14.514288 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:09:14.514301 (+ 13us) server_negotiation.cc:407] Connection header received
0924 04:09:14.514391 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:09:14.514393 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:09:14.514435 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:09:14.514512 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:09:14.514926 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:14.515451 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:14.516196 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:14.516428 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:14.516506 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:14.519005 (+ 2499us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:44495: BlockingRecv error: recv got EOF from 127.0.98.1:44495 (error 108)
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":119,"threads_started":1}
Sep 24 04:09:14 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:09:14 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:09:14.707719 9592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:09:14.702611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42249 (local address 127.0.98.2:53291)
0924 04:09:14.702870 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0924 04:09:14.702883 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:09:14.703009 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:09:14.703236 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:09:14.703238 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:09:14.703413 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:09:14.703508 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:14.703513 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:14.704240 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:14.704251 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:14.704838 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:14.704847 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:14.705256 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:14.705259 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:14.705318 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:14.705798 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:09:14.705809 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:09:14.707587 (+ 1778us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42249: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":98,"threads_started":1}
W20260924 04:09:14.707800 9591 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:09:14.702729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53291 (local address 127.0.98.62:42249)
0924 04:09:14.702863 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 04:09:14.702866 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:09:14.702951 (+ 85us) server_negotiation.cc:407] Connection header received
0924 04:09:14.703089 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:09:14.703091 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:09:14.703133 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:09:14.703201 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:09:14.703605 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:14.704123 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:14.704937 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:14.705159 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:14.705242 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:14.707709 (+ 2467us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:53291: BlockingRecv error: recv got EOF from 127.0.98.2:53291 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 24 04:09:14 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:09:14 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:09:14.890820 9593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:09:14.885087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42249 (local address 127.0.98.3:34533)
0924 04:09:14.885407 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0924 04:09:14.885425 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 04:09:14.885569 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:09:14.885863 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:09:14.885866 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:09:14.886093 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:09:14.886182 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:14.886188 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:14.886995 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:14.887000 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:14.887732 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:14.887741 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:14.888286 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:14.888290 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:14.888372 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:14.888852 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:09:14.888862 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:09:14.890648 (+ 1786us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42249: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":101,"threads_started":1}
W20260924 04:09:14.890879 9591 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:09:14.885219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34533 (local address 127.0.98.62:42249)
0924 04:09:14.885394 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 04:09:14.885399 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:09:14.885532 (+ 133us) server_negotiation.cc:407] Connection header received
0924 04:09:14.885679 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:09:14.885682 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:09:14.885749 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:09:14.885843 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:09:14.886305 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:14.886886 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:14.887842 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:14.888176 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:14.888285 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:14.890772 (+ 2487us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:34533: BlockingRecv error: recv got EOF from 127.0.98.3:34533 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 24 04:09:15 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:09:15 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:09:15.525625 9594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:09:15.519851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42249 (local address 127.0.98.1:36779)
0924 04:09:15.520138 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0924 04:09:15.520152 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:09:15.520271 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:09:15.520604 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:09:15.520610 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:09:15.520878 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:09:15.521022 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:15.521030 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:15.521996 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:15.521999 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:15.522549 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:15.522556 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:15.523011 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:15.523014 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:15.523077 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:15.523602 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:09:15.523613 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:09:15.525514 (+ 1901us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42249: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":110,"threads_started":1}
W20260924 04:09:15.525831 9595 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:09:15.519985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36779 (local address 127.0.98.62:42249)
0924 04:09:15.520357 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0924 04:09:15.520361 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:09:15.520375 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:09:15.520414 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:09:15.520416 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:09:15.520456 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:09:15.520547 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:09:15.521185 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:15.521893 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:15.522656 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:15.522896 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:15.522978 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:15.525654 (+ 2676us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:36779: BlockingRecv error: recv got EOF from 127.0.98.1:36779 (error 108)
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":192,"threads_started":1}
Sep 24 04:09:15 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:09:15 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:09:15.714097 9596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:09:15.708611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42249 (local address 127.0.98.2:57873)
0924 04:09:15.708979 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0924 04:09:15.708992 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:09:15.709103 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:09:15.709328 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:09:15.709330 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:09:15.709525 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:09:15.709619 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:15.709624 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:15.710418 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:15.710421 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:15.711008 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:15.711017 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:15.711473 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:15.711477 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:15.711546 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:15.712069 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:09:15.712080 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:09:15.713980 (+ 1900us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42249: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":185,"threads_started":1}
W20260924 04:09:15.714290 9595 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:09:15.708714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57873 (local address 127.0.98.62:42249)
0924 04:09:15.708891 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 04:09:15.708894 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:09:15.709166 (+ 272us) server_negotiation.cc:407] Connection header received
0924 04:09:15.709209 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:09:15.709211 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:09:15.709256 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:09:15.709314 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:09:15.709747 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:15.710314 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:15.711121 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:15.711347 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:15.711432 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:15.714146 (+ 2714us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:57873: BlockingRecv error: recv got EOF from 127.0.98.2:57873 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Sep 24 04:09:15 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:09:15 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:09:15.897773 9597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:09:15.891675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42249 (local address 127.0.98.3:36143)
0924 04:09:15.892126 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0924 04:09:15.892142 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:09:15.892298 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:09:15.892591 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:09:15.892594 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:09:15.892829 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:09:15.892918 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:15.892924 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:15.893695 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:15.893701 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:15.894478 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:15.894504 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:15.895133 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:15.895137 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:15.895219 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:15.895748 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:09:15.895769 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:09:15.897660 (+ 1891us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42249: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":100,"threads_started":1}
W20260924 04:09:15.897961 9595 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:09:15.891822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36143 (local address 127.0.98.62:42249)
0924 04:09:15.892118 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0924 04:09:15.892123 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:09:15.892232 (+ 109us) server_negotiation.cc:407] Connection header received
0924 04:09:15.892421 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:09:15.892424 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:09:15.892478 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:09:15.892568 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:09:15.893044 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:15.893582 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:15.894636 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:15.894954 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:15.895084 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:15.897800 (+ 2716us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:36143: BlockingRecv error: recv got EOF from 127.0.98.3:36143 (error 108)
Metrics: {"server-negotiator.queue_time_us":162}
Sep 24 04:09:16 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:09:16 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:09:16.532020 9598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:09:16.526330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42249 (local address 127.0.98.1:50381)
0924 04:09:16.526615 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0924 04:09:16.526629 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:09:16.526724 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:09:16.527045 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:09:16.527048 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:09:16.527231 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:09:16.527328 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:16.527333 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:16.528084 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:16.528086 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:16.528631 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:16.528638 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:16.529356 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:16.529359 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:16.529431 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:16.530024 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:09:16.530036 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:09:16.531839 (+ 1803us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42249: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":72,"threads_started":1}
W20260924 04:09:16.532238 9599 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:09:16.526591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50381 (local address 127.0.98.62:42249)
0924 04:09:16.526804 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0924 04:09:16.526808 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:09:16.526819 (+ 11us) server_negotiation.cc:407] Connection header received
0924 04:09:16.526859 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:09:16.526861 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:09:16.526902 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:09:16.526978 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:09:16.527429 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:16.527987 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:16.528973 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:16.529211 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:16.529300 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:16.532050 (+ 2750us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:50381: BlockingRecv error: recv got EOF from 127.0.98.1:50381 (error 108)
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":69,"threads_started":1}
Sep 24 04:09:16 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:09:16 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:09:16.720762 9600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:09:16.714904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42249 (local address 127.0.98.2:56217)
0924 04:09:16.715177 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0924 04:09:16.715191 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:09:16.715276 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:09:16.715548 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:09:16.715550 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:09:16.715763 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:09:16.715901 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:16.715908 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:16.716821 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:16.716824 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:16.717420 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:16.717429 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:16.717896 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:16.717899 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:16.717969 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:16.718514 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:09:16.718525 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:09:16.720563 (+ 2038us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42249: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":109,"threads_started":1}
W20260924 04:09:16.720921 9599 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:09:16.715095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56217 (local address 127.0.98.62:42249)
0924 04:09:16.715230 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 04:09:16.715233 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:09:16.715246 (+ 13us) server_negotiation.cc:407] Connection header received
0924 04:09:16.715361 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:09:16.715363 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:09:16.715405 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:09:16.715476 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:09:16.716030 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:16.716620 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:16.717538 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:16.717762 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:16.717860 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:16.720803 (+ 2943us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:56217: BlockingRecv error: recv got EOF from 127.0.98.2:56217 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 24 04:09:16 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:09:16 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:09:16.904384 9601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:09:16.898488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42249 (local address 127.0.98.3:46733)
0924 04:09:16.898765 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0924 04:09:16.898777 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:09:16.898872 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:09:16.899126 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:09:16.899129 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:09:16.899303 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:09:16.899402 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:16.899406 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:16.900726 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:16.900729 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:16.901287 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:16.901296 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:16.901752 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:16.901754 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:16.901813 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:16.902269 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:09:16.902280 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:09:16.904220 (+ 1940us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42249: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":114,"threads_started":1}
W20260924 04:09:16.904836 9599 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:09:16.898570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46733 (local address 127.0.98.62:42249)
0924 04:09:16.898698 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 04:09:16.898705 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 04:09:16.898844 (+ 139us) server_negotiation.cc:407] Connection header received
0924 04:09:16.898965 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:09:16.898967 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:09:16.899015 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:09:16.899087 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:09:16.899576 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:16.900572 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:16.901414 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:16.901643 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:16.901739 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:16.904700 (+ 2961us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:46733: BlockingRecv error: recv got EOF from 127.0.98.3:46733 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 24 04:09:17 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:09:17 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:09:17.538663 9602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:09:17.532893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42249 (local address 127.0.98.1:41847)
0924 04:09:17.533166 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0924 04:09:17.533223 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 04:09:17.533335 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:09:17.533619 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:09:17.533621 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:09:17.533846 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:09:17.533941 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:17.533947 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:17.534880 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:17.534882 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:17.535446 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:17.535454 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:17.535989 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:17.535992 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:17.536063 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:17.536572 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:09:17.536583 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:09:17.538539 (+ 1956us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42249: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":91,"threads_started":1}
W20260924 04:09:17.538806 9603 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:09:17.532961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41847 (local address 127.0.98.62:42249)
0924 04:09:17.533242 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0924 04:09:17.533245 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:09:17.533283 (+ 38us) server_negotiation.cc:407] Connection header received
0924 04:09:17.533426 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:09:17.533429 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:09:17.533496 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:09:17.533583 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:09:17.534067 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:17.534764 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:17.535547 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:17.535772 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:17.535847 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:17.538675 (+ 2828us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:41847: BlockingRecv error: recv got EOF from 127.0.98.1:41847 (error 108)
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":79,"threads_started":1}
Sep 24 04:09:17 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:09:17 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:09:17.726963 9604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:09:17.721682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42249 (local address 127.0.98.2:52139)
0924 04:09:17.721961 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0924 04:09:17.721974 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:09:17.722070 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:09:17.722326 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:09:17.722329 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:09:17.722513 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:09:17.722614 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:17.722619 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:17.723360 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:17.723363 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:17.723961 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:17.723970 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:17.724381 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:17.724384 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:17.724448 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:17.724959 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:09:17.724970 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:09:17.726830 (+ 1860us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42249: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":83,"threads_started":1}
W20260924 04:09:17.727088 9603 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:09:17.721756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52139 (local address 127.0.98.62:42249)
0924 04:09:17.721922 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 04:09:17.721926 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:09:17.722023 (+ 97us) server_negotiation.cc:407] Connection header received
0924 04:09:17.722134 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:09:17.722136 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:09:17.722176 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:09:17.722245 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:09:17.722729 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:17.723256 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:17.724066 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:17.724283 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:17.724356 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:17.726973 (+ 2617us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:52139: BlockingRecv error: recv got EOF from 127.0.98.2:52139 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 24 04:09:17 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:09:17 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:09:17.912068 9605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:09:17.905247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42249 (local address 127.0.98.3:40931)
0924 04:09:17.905531 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0924 04:09:17.905544 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:09:17.905645 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:09:17.906007 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:09:17.906012 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:09:17.906239 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:09:17.906348 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:17.906354 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:17.907296 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:17.907302 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:17.908197 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:17.908210 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:17.908856 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:17.908862 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:17.909020 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:17.909614 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:09:17.909635 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:09:17.911930 (+ 2295us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42249: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":112,"threads_started":1}
W20260924 04:09:17.912158 9603 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:09:17.905377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40931 (local address 127.0.98.62:42249)
0924 04:09:17.905509 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 04:09:17.905513 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:09:17.905621 (+ 108us) server_negotiation.cc:407] Connection header received
0924 04:09:17.905792 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:09:17.905796 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:09:17.905857 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:09:17.905956 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:09:17.906508 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:17.907175 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:17.908334 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:17.908666 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:17.908831 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:17.912064 (+ 3233us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:40931: BlockingRecv error: recv got EOF from 127.0.98.3:40931 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 24 04:09:18 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:09:18 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:09:18.551869 9606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:09:18.539701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42249 (local address 127.0.98.1:39985)
0924 04:09:18.539978 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0924 04:09:18.539995 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 04:09:18.540093 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:09:18.540374 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:09:18.540376 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:09:18.540550 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:09:18.540640 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:18.540644 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:18.541456 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:18.541458 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:18.542009 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:18.542017 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:18.542455 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:18.542458 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:18.542515 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:18.549887 (+ 7372us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:09:18.549901 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:09:18.551737 (+ 1836us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42249: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":72,"threads_started":1}
W20260924 04:09:18.552112 9607 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:09:18.539835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39985 (local address 127.0.98.62:42249)
0924 04:09:18.540130 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0924 04:09:18.540134 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:09:18.540147 (+ 13us) server_negotiation.cc:407] Connection header received
0924 04:09:18.540184 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:09:18.540187 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:09:18.540226 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:09:18.540317 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:09:18.540826 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:18.541359 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:18.542116 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:18.542343 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:18.542427 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:18.551960 (+ 9533us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:39985: BlockingRecv error: recv got EOF from 127.0.98.1:39985 (error 108)
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":75,"threads_started":1}
Sep 24 04:09:18 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:09:18 dist-test-slave-00ld krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:09:18.734129 9608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:09:18.728032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42249 (local address 127.0.98.2:54401)
0924 04:09:18.728363 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0924 04:09:18.728380 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 04:09:18.728495 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:09:18.728838 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:09:18.728841 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:09:18.729056 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:09:18.729146 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:18.729151 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:18.729932 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:18.729937 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:18.730525 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:18.730533 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:18.731005 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:18.731009 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:18.731096 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:18.731721 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:09:18.731739 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:09:18.734009 (+ 2270us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42249: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":93,"threads_started":1}
W20260924 04:09:18.734305 9607 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:09:18.728210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54401 (local address 127.0.98.62:42249)
0924 04:09:18.728367 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 04:09:18.728374 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 04:09:18.728444 (+ 70us) server_negotiation.cc:407] Connection header received
0924 04:09:18.728609 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:09:18.728612 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:09:18.728668 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:09:18.728816 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:09:18.729276 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:18.729818 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:18.730652 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:18.730896 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:18.731002 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:18.734172 (+ 3170us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:54401: BlockingRecv error: recv got EOF from 127.0.98.2:54401 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 04:09:18.813458 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 9132
I20260924 04:09:18.819235 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 9284
I20260924 04:09:18.824482 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 9416
I20260924 04:09:18.829510 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 9062
2026-09-24T04:09:18Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17980 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-24T04:09:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:09:18Z Disabled control of system clock
I20260924 04:09:18.859666 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222898807720-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222898807720-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222898807720-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38327
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:33081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38327
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 04:09:18.970155 9612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:09:18.970433 9612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:09:18.970466 9612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:09:18.974467 9612 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:09:18.974530 9612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:09:18.974546 9612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:09:18.974563 9612 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:09:18.974578 9612 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:09:18.979447 9612 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222898807720-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222898807720-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38327
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38327
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9612
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222898807720-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:09:18.980535 9612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:09:18.981995 9612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:09:18.990878 9617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:09:18.990863 9612 server_base.cc:1034] running on GCE node
W20260924 04:09:18.990972 9620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:09:18.990861 9618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:09:18.991676 9612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:09:18.992815 9612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:09:18.993976 9612 hybrid_clock.cc:648] HybridClock initialized: now 1790222958993962 us; error 33 us; skew 500 ppm
I20260924 04:09:18.995698 9612 webserver.cc:460] Webserver started at http://127.0.98.62:43037/ using document root <none> and password file <none>
I20260924 04:09:18.996217 9612 fs_manager.cc:359] Metadata directory not provided
I20260924 04:09:18.996275 9612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:09:18.996441 9612 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:09:18.998471 9612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222898807720-392-0/minicluster-data/master-0/data/instance:
uuid: "96fb7071ca0a48e8bc8738ad722e6978"
format_stamp: "Formatted at 2026-09-24 04:09:18 on dist-test-slave-00ld"
server_key: "9fc611079f38cf559a0e308c2b9e2749"
server_key_iv: "7b3dd2c2ae4bfebbb5f5668366a5cabd"
server_key_version: "encryptionkey@0"
I20260924 04:09:18.998978 9612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222898807720-392-0/minicluster-data/master-0/wal/instance:
uuid: "96fb7071ca0a48e8bc8738ad722e6978"
format_stamp: "Formatted at 2026-09-24 04:09:18 on dist-test-slave-00ld"
server_key: "9fc611079f38cf559a0e308c2b9e2749"
server_key_iv: "7b3dd2c2ae4bfebbb5f5668366a5cabd"
server_key_version: "encryptionkey@0"
I20260924 04:09:19.002673 9612 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 04:09:19.005101 9626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:09:19.006290 9612 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:09:19.006458 9612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222898807720-392-0/minicluster-data/master-0/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222898807720-392-0/minicluster-data/master-0/wal
uuid: "96fb7071ca0a48e8bc8738ad722e6978"
format_stamp: "Formatted at 2026-09-24 04:09:18 on dist-test-slave-00ld"
server_key: "9fc611079f38cf559a0e308c2b9e2749"
server_key_iv: "7b3dd2c2ae4bfebbb5f5668366a5cabd"
server_key_version: "encryptionkey@0"
I20260924 04:09:19.006568 9612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222898807720-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222898807720-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222898807720-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:09:19.021536 9612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:09:19.022301 9612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:09:19.022511 9612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:09:19.030082 9612 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38327
I20260924 04:09:19.030090 9678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38327 every 8 connection(s)
I20260924 04:09:19.031060 9612 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:09:19.034319 9679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:09:19.035415 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 9612
I20260924 04:09:19.035516 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222898807720-392-0/minicluster-data/master-0/wal/instance
I20260924 04:09:19.035832 392 external_mini_cluster.cc:1428] Setting key b5ec3b2db512e57fb0241aa601b40d63
I20260924 04:09:19.041000 9679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96fb7071ca0a48e8bc8738ad722e6978: Bootstrap starting.
I20260924 04:09:19.043797 9679 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 96fb7071ca0a48e8bc8738ad722e6978: Neither blocks nor log segments found. Creating new log.
I20260924 04:09:19.044770 9679 log.cc:824] T 00000000000000000000000000000000 P 96fb7071ca0a48e8bc8738ad722e6978: Log is configured to *not* fsync() on all Append() calls
I20260924 04:09:19.046525 9682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:09:19.037331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53832 (local address 127.0.98.62:38327)
0924 04:09:19.038028 (+ 697us) server_negotiation.cc:206] Beginning negotiation
0924 04:09:19.038048 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 04:09:19.038091 (+ 43us) server_negotiation.cc:407] Connection header received
0924 04:09:19.038943 (+ 852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:09:19.038966 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:09:19.039489 (+ 523us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:09:19.039870 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:09:19.040326 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:19.041765 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:19.042699 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:19.043386 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:19.043555 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:19.044348 (+ 793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:09:19.044354 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:09:19.044371 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:09:19.044928 (+ 557us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:09:19.044997 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:09:19.045023 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:09:19.045128 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 04:09:19.045280 (+ 152us) server_negotiation.cc:299] Negotiation successful
0924 04:09:19.045606 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":121,"threads_started":1}
I20260924 04:09:19.047045 9679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96fb7071ca0a48e8bc8738ad722e6978: No bootstrap required, opened a new log
I20260924 04:09:19.050789 9679 raft_consensus.cc:348] T 00000000000000000000000000000000 P 96fb7071ca0a48e8bc8738ad722e6978 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96fb7071ca0a48e8bc8738ad722e6978" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38327 } }
I20260924 04:09:19.051014 9679 raft_consensus.cc:374] T 00000000000000000000000000000000 P 96fb7071ca0a48e8bc8738ad722e6978 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:09:19.051049 9679 raft_consensus.cc:729] T 00000000000000000000000000000000 P 96fb7071ca0a48e8bc8738ad722e6978 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96fb7071ca0a48e8bc8738ad722e6978, State: Initialized, Role: FOLLOWER
I20260924 04:09:19.051414 9679 consensus_queue.cc:260] T 00000000000000000000000000000000 P 96fb7071ca0a48e8bc8738ad722e6978 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96fb7071ca0a48e8bc8738ad722e6978" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38327 } }
I20260924 04:09:19.051520 9679 raft_consensus.cc:388] T 00000000000000000000000000000000 P 96fb7071ca0a48e8bc8738ad722e6978 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:09:19.051568 9679 raft_consensus.cc:482] T 00000000000000000000000000000000 P 96fb7071ca0a48e8bc8738ad722e6978 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:09:19.051637 9679 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 96fb7071ca0a48e8bc8738ad722e6978 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:09:19.052522 9679 raft_consensus.cc:504] T 00000000000000000000000000000000 P 96fb7071ca0a48e8bc8738ad722e6978 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96fb7071ca0a48e8bc8738ad722e6978" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38327 } }
I20260924 04:09:19.052898 9679 leader_election.cc:302] T 00000000000000000000000000000000 P 96fb7071ca0a48e8bc8738ad722e6978 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 96fb7071ca0a48e8bc8738ad722e6978; no voters:
I20260924 04:09:19.053227 9679 leader_election.cc:288] T 00000000000000000000000000000000 P 96fb7071ca0a48e8bc8738ad722e6978 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:09:19.053354 9684 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 96fb7071ca0a48e8bc8738ad722e6978 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:09:19.053583 9684 raft_consensus.cc:686] T 00000000000000000000000000000000 P 96fb7071ca0a48e8bc8738ad722e6978 [term 1 LEADER]: Becoming Leader. State: Replica: 96fb7071ca0a48e8bc8738ad722e6978, State: Running, Role: LEADER
I20260924 04:09:19.053939 9684 consensus_queue.cc:237] T 00000000000000000000000000000000 P 96fb7071ca0a48e8bc8738ad722e6978 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96fb7071ca0a48e8bc8738ad722e6978" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38327 } }
I20260924 04:09:19.054507 9679 sys_catalog.cc:564] T 00000000000000000000000000000000 P 96fb7071ca0a48e8bc8738ad722e6978 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:09:19.055684 9685 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96fb7071ca0a48e8bc8738ad722e6978 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "96fb7071ca0a48e8bc8738ad722e6978" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96fb7071ca0a48e8bc8738ad722e6978" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38327 } } }
I20260924 04:09:19.055706 9686 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96fb7071ca0a48e8bc8738ad722e6978 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 96fb7071ca0a48e8bc8738ad722e6978. Latest consensus state: current_term: 1 leader_uuid: "96fb7071ca0a48e8bc8738ad722e6978" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96fb7071ca0a48e8bc8738ad722e6978" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38327 } } }
I20260924 04:09:19.055804 9685 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96fb7071ca0a48e8bc8738ad722e6978 [sys.catalog]: This master's current role is: LEADER
I20260924 04:09:19.055811 9686 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96fb7071ca0a48e8bc8738ad722e6978 [sys.catalog]: This master's current role is: LEADER
I20260924 04:09:19.057015 9693 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:09:19.059724 9693 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:09:19.065737 9693 catalog_manager.cc:1347] Generated new cluster ID: 942545355c554270a6d62bb4e6854523
I20260924 04:09:19.065830 9693 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:09:19.071745 9693 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:09:19.072991 9693 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:09:19.085853 9693 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 96fb7071ca0a48e8bc8738ad722e6978: Generated new TSK 0
I20260924 04:09:19.086683 9693 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:09:19.095127 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222898807720-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222898807720-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222898807720-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222898807720-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38327
--builtin_ntp_servers=127.0.98.20:33081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260924 04:09:19.208418 9703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:09:19.208671 9703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:09:19.208772 9703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:09:19.213441 9703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:09:19.213505 9703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:09:19.213560 9703 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:09:19.218488 9703 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222898807720-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222898807720-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222898807720-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38327
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9703
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222898807720-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:09:19.219523 9703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:09:19.220943 9703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:09:19.230300 9708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:09:19.230481 9703 server_base.cc:1034] running on GCE node
W20260924 04:09:19.230358 9711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:09:19.230304 9709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:09:19.231060 9703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:09:19.231729 9703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:09:19.232913 9703 hybrid_clock.cc:648] HybridClock initialized: now 1790222959232898 us; error 26 us; skew 500 ppm
I20260924 04:09:19.234730 9703 webserver.cc:460] Webserver started at http://127.0.98.1:45925/ using document root <none> and password file <none>
I20260924 04:09:19.235296 9703 fs_manager.cc:359] Metadata directory not provided
I20260924 04:09:19.235394 9703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:09:19.235633 9703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:09:19.237699 9703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222898807720-392-0/minicluster-data/ts-0/data/instance:
uuid: "3fc36e3b12a746d3ae3c19e274a40fc1"
format_stamp: "Formatted at 2026-09-24 04:09:19 on dist-test-slave-00ld"
server_key: "b9738c02e7b48da7f83a12e05db78387"
server_key_iv: "43ac64c20dc3ea5f21c46ffd8e691bc2"
server_key_version: "encryptionkey@0"
I20260924 04:09:19.238240 9703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222898807720-392-0/minicluster-data/ts-0/wal/instance:
uuid: "3fc36e3b12a746d3ae3c19e274a40fc1"
format_stamp: "Formatted at 2026-09-24 04:09:19 on dist-test-slave-00ld"
server_key: "b9738c02e7b48da7f83a12e05db78387"
server_key_iv: "43ac64c20dc3ea5f21c46ffd8e691bc2"
server_key_version: "encryptionkey@0"
I20260924 04:09:19.241808 9703 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260924 04:09:19.244392 9717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:09:19.245702 9703 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 04:09:19.245877 9703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222898807720-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222898807720-392-0/minicluster-data/ts-0/wal
uuid: "3fc36e3b12a746d3ae3c19e274a40fc1"
format_stamp: "Formatted at 2026-09-24 04:09:19 on dist-test-slave-00ld"
server_key: "b9738c02e7b48da7f83a12e05db78387"
server_key_iv: "43ac64c20dc3ea5f21c46ffd8e691bc2"
server_key_version: "encryptionkey@0"
I20260924 04:09:19.245998 9703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222898807720-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222898807720-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222898807720-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:09:19.261685 9703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:09:19.262444 9703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:09:19.262655 9703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:09:19.263263 9703 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:09:19.264228 9703 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:09:19.264297 9703 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:09:19.264371 9703 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:09:19.264405 9703 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:09:19.272984 9703 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39971
I20260924 04:09:19.273090 9830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39971 every 8 connection(s)
I20260924 04:09:19.274150 9703 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222898807720-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:09:19.281288 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 9703
I20260924 04:09:19.281386 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222898807720-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:09:19.281656 392 external_mini_cluster.cc:1428] Setting key 9359a628cd9ea78dd21038ca779da9ad
I20260924 04:09:19.283478 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222898807720-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222898807720-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222898807720-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222898807720-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38327
--builtin_ntp_servers=127.0.98.20:33081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260924 04:09:19.284005 9682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:09:19.276065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50569 (local address 127.0.98.62:38327)
0924 04:09:19.276207 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 04:09:19.276211 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:09:19.277426 (+ 1215us) server_negotiation.cc:407] Connection header received
0924 04:09:19.278919 (+ 1493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:09:19.278922 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:09:19.278978 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:09:19.279063 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:09:19.279972 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:19.280529 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:19.281834 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:19.282079 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:19.282202 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:19.283316 (+ 1114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:09:19.283319 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:09:19.283322 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:09:19.283334 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:09:19.283360 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:09:19.283368 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:09:19.283445 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 04:09:19.283712 (+ 267us) server_negotiation.cc:299] Negotiation successful
0924 04:09:19.283781 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 04:09:19.284777 9833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:09:19.276485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38327 (local address 127.0.98.1:50569)
0924 04:09:19.277149 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0924 04:09:19.277196 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 04:09:19.278701 (+ 1505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:09:19.279235 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:09:19.279242 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:09:19.279326 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:09:19.279826 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:19.279838 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:19.280674 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:19.280718 (+ 44us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:19.281681 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:19.281714 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:19.282240 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:19.282244 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:19.282419 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:19.283043 (+ 624us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:09:19.283045 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:09:19.283084 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:09:19.283158 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:09:19.283448 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:09:19.283459 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:09:19.283466 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 04:09:19.283633 (+ 167us) client_negotiation.cc:240] Negotiation successful
0924 04:09:19.283882 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":112,"threads_started":1}
I20260924 04:09:19.286157 9831 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38327
I20260924 04:09:19.286392 9831 heartbeater.cc:464] Registering TS with master...
I20260924 04:09:19.286976 9831 heartbeater.cc:511] Master 127.0.98.62:38327 requested a full tablet report, sending...
I20260924 04:09:19.288515 9643 ts_manager.cc:194] Registered new tserver with Master: 3fc36e3b12a746d3ae3c19e274a40fc1 (127.0.98.1:39971)
I20260924 04:09:19.289707 9643 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:50569
W20260924 04:09:19.403509 9834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:09:19.403826 9834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:09:19.403903 9834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:09:19.408232 9834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:09:19.408335 9834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:09:19.408429 9834 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:09:19.413573 9834 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222898807720-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222898807720-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222898807720-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38327
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9834
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222898807720-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:09:19.414691 9834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:09:19.416115 9834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:09:19.425808 9839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:09:19.425778 9842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:09:19.425886 9834 server_base.cc:1034] running on GCE node
W20260924 04:09:19.425793 9840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:09:19.426502 9834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:09:19.427181 9834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:09:19.428376 9834 hybrid_clock.cc:648] HybridClock initialized: now 1790222959428364 us; error 27 us; skew 500 ppm
I20260924 04:09:19.430341 9834 webserver.cc:460] Webserver started at http://127.0.98.2:42991/ using document root <none> and password file <none>
I20260924 04:09:19.430915 9834 fs_manager.cc:359] Metadata directory not provided
I20260924 04:09:19.431007 9834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:09:19.431252 9834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:09:19.433391 9834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222898807720-392-0/minicluster-data/ts-1/data/instance:
uuid: "29bd3732a8574cac8793eb5480aa0cc7"
format_stamp: "Formatted at 2026-09-24 04:09:19 on dist-test-slave-00ld"
server_key: "d9f3198000aea665352aec37db98781d"
server_key_iv: "baba96b3e29f4a8c40e98e5ec16969e6"
server_key_version: "encryptionkey@0"
I20260924 04:09:19.433954 9834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222898807720-392-0/minicluster-data/ts-1/wal/instance:
uuid: "29bd3732a8574cac8793eb5480aa0cc7"
format_stamp: "Formatted at 2026-09-24 04:09:19 on dist-test-slave-00ld"
server_key: "d9f3198000aea665352aec37db98781d"
server_key_iv: "baba96b3e29f4a8c40e98e5ec16969e6"
server_key_version: "encryptionkey@0"
I20260924 04:09:19.437690 9834 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.001s
I20260924 04:09:19.440248 9848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:09:19.441593 9834 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:09:19.441741 9834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222898807720-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222898807720-392-0/minicluster-data/ts-1/wal
uuid: "29bd3732a8574cac8793eb5480aa0cc7"
format_stamp: "Formatted at 2026-09-24 04:09:19 on dist-test-slave-00ld"
server_key: "d9f3198000aea665352aec37db98781d"
server_key_iv: "baba96b3e29f4a8c40e98e5ec16969e6"
server_key_version: "encryptionkey@0"
I20260924 04:09:19.441866 9834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222898807720-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222898807720-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222898807720-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:09:19.459498 9834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:09:19.460315 9834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:09:19.460513 9834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:09:19.461186 9834 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:09:19.462173 9834 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:09:19.462262 9834 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:09:19.462363 9834 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:09:19.462419 9834 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:09:19.471467 9834 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46479
I20260924 04:09:19.471493 9961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46479 every 8 connection(s)
I20260924 04:09:19.472384 9834 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222898807720-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:09:19.480633 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 9834
I20260924 04:09:19.480774 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222898807720-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:09:19.481082 392 external_mini_cluster.cc:1428] Setting key f3d933aa2a848c4f1f00c61df1b25237
I20260924 04:09:19.481657 9682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:09:19.474214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35769 (local address 127.0.98.62:38327)
0924 04:09:19.474388 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 04:09:19.474393 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:09:19.475556 (+ 1163us) server_negotiation.cc:407] Connection header received
0924 04:09:19.476996 (+ 1440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:09:19.476999 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:09:19.477044 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:09:19.477121 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:09:19.478013 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:19.478535 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:19.479611 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:19.479851 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:19.479951 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:19.481187 (+ 1236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:09:19.481190 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:09:19.481193 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:09:19.481210 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:09:19.481231 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:09:19.481242 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:09:19.481328 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 04:09:19.481477 (+ 149us) server_negotiation.cc:299] Negotiation successful
0924 04:09:19.481546 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 04:09:19.482780 9964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:09:19.474641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38327 (local address 127.0.98.2:35769)
0924 04:09:19.475354 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0924 04:09:19.475436 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0924 04:09:19.476808 (+ 1372us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:09:19.477281 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:09:19.477288 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:09:19.477371 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:09:19.477881 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:19.477893 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:19.478632 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:19.478635 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:19.479506 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:19.479515 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:19.480026 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:19.480029 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:19.480168 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:19.480926 (+ 758us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:09:19.480928 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:09:19.480969 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:09:19.481033 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:09:19.481338 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:09:19.481345 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:09:19.481349 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 04:09:19.481443 (+ 94us) client_negotiation.cc:240] Negotiation successful
0924 04:09:19.481748 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":110,"threads_started":1}
I20260924 04:09:19.483186 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222898807720-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222898807720-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222898807720-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222898807720-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38327
--builtin_ntp_servers=127.0.98.20:33081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260924 04:09:19.483882 9962 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38327
I20260924 04:09:19.484126 9962 heartbeater.cc:464] Registering TS with master...
I20260924 04:09:19.484783 9962 heartbeater.cc:511] Master 127.0.98.62:38327 requested a full tablet report, sending...
I20260924 04:09:19.485858 9643 ts_manager.cc:194] Registered new tserver with Master: 29bd3732a8574cac8793eb5480aa0cc7 (127.0.98.2:46479)
I20260924 04:09:19.486423 9643 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:35769
W20260924 04:09:19.593631 9965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:09:19.593905 9965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:09:19.594003 9965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:09:19.598016 9965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:09:19.598093 9965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:09:19.598172 9965 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:09:19.602880 9965 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222898807720-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222898807720-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222898807720-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38327
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9965
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222898807720-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:09:19.603957 9965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:09:19.605387 9965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:09:19.615142 9970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:09:19.615113 9973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:09:19.615136 9971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:09:19.615409 9965 server_base.cc:1034] running on GCE node
I20260924 04:09:19.615805 9965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:09:19.616441 9965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:09:19.617615 9965 hybrid_clock.cc:648] HybridClock initialized: now 1790222959617585 us; error 41 us; skew 500 ppm
I20260924 04:09:19.619421 9965 webserver.cc:460] Webserver started at http://127.0.98.3:45091/ using document root <none> and password file <none>
I20260924 04:09:19.619988 9965 fs_manager.cc:359] Metadata directory not provided
I20260924 04:09:19.620075 9965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:09:19.620298 9965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:09:19.622329 9965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222898807720-392-0/minicluster-data/ts-2/data/instance:
uuid: "79b378c6af444f25a177abc9f5f7c993"
format_stamp: "Formatted at 2026-09-24 04:09:19 on dist-test-slave-00ld"
server_key: "a7b39fda0e5ba28d030b5b83b191ebf4"
server_key_iv: "ea7d8bb9cadac3d3ed566e60aa7921a3"
server_key_version: "encryptionkey@0"
I20260924 04:09:19.622889 9965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222898807720-392-0/minicluster-data/ts-2/wal/instance:
uuid: "79b378c6af444f25a177abc9f5f7c993"
format_stamp: "Formatted at 2026-09-24 04:09:19 on dist-test-slave-00ld"
server_key: "a7b39fda0e5ba28d030b5b83b191ebf4"
server_key_iv: "ea7d8bb9cadac3d3ed566e60aa7921a3"
server_key_version: "encryptionkey@0"
I20260924 04:09:19.626375 9965 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260924 04:09:19.628839 9979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:09:19.629997 9965 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 04:09:19.630127 9965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222898807720-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222898807720-392-0/minicluster-data/ts-2/wal
uuid: "79b378c6af444f25a177abc9f5f7c993"
format_stamp: "Formatted at 2026-09-24 04:09:19 on dist-test-slave-00ld"
server_key: "a7b39fda0e5ba28d030b5b83b191ebf4"
server_key_iv: "ea7d8bb9cadac3d3ed566e60aa7921a3"
server_key_version: "encryptionkey@0"
I20260924 04:09:19.630232 9965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222898807720-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222898807720-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222898807720-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:09:19.644513 9965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:09:19.645347 9965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:09:19.645532 9965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:09:19.646147 9965 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:09:19.647073 9965 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:09:19.647140 9965 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:09:19.647205 9965 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:09:19.647235 9965 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:09:19.656397 9965 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39617
I20260924 04:09:19.656487 10092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39617 every 8 connection(s)
I20260924 04:09:19.657421 9965 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222898807720-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:09:19.659585 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 9965
I20260924 04:09:19.659667 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790222898807720-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:09:19.659965 392 external_mini_cluster.cc:1428] Setting key 8d99b5f0247188a7292171a99bbbc1de
I20260924 04:09:19.666805 9682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:09:19.659205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50673 (local address 127.0.98.62:38327)
0924 04:09:19.659377 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 04:09:19.659382 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:09:19.660569 (+ 1187us) server_negotiation.cc:407] Connection header received
0924 04:09:19.662193 (+ 1624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:09:19.662196 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:09:19.662251 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:09:19.662327 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:09:19.663217 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:19.663751 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:19.664962 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:19.665191 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:19.665299 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:19.666308 (+ 1009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:09:19.666312 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:09:19.666315 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:09:19.666327 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:09:19.666347 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:09:19.666356 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:09:19.666434 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 04:09:19.666600 (+ 166us) server_negotiation.cc:299] Negotiation successful
0924 04:09:19.666657 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 04:09:19.667583 10095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:09:19.659647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38327 (local address 127.0.98.3:50673)
0924 04:09:19.660414 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0924 04:09:19.660465 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 04:09:19.661976 (+ 1511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:09:19.662517 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:09:19.662524 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:09:19.662609 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:09:19.663087 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:19.663098 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:19.663877 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:19.663881 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:19.664834 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:19.664843 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:19.665337 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:19.665341 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:19.665515 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:19.666065 (+ 550us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:09:19.666067 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:09:19.666105 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:09:19.666163 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:09:19.666447 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:09:19.666456 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:09:19.666462 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 04:09:19.666589 (+ 127us) client_negotiation.cc:240] Negotiation successful
0924 04:09:19.666849 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":121,"threads_started":1}
I20260924 04:09:19.668637 10093 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38327
I20260924 04:09:19.668920 10093 heartbeater.cc:464] Registering TS with master...
I20260924 04:09:19.669447 10093 heartbeater.cc:511] Master 127.0.98.62:38327 requested a full tablet report, sending...
I20260924 04:09:19.670425 9643 ts_manager.cc:194] Registered new tserver with Master: 79b378c6af444f25a177abc9f5f7c993 (127.0.98.3:39617)
I20260924 04:09:19.670938 9643 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:50673
I20260924 04:09:19.673722 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 04:09:19.676509 9682 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:09:19.675430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34856 (local address 127.0.98.62:38327)
0924 04:09:19.675536 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0924 04:09:19.675539 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:09:19.675864 (+ 325us) server_negotiation.cc:407] Connection header received
0924 04:09:19.676013 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:09:19.676016 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:09:19.676063 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:09:19.676151 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:09:19.676399 (+ 248us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34856: BlockingRecv error: recv got EOF from 127.0.0.1:34856 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 04:09:19.676699 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 9703
I20260924 04:09:19.683534 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 9834
I20260924 04:09:19.688781 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 9965
I20260924 04:09:19.693771 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 9612
2026-09-24T04:09:19Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (862 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-24T04:09:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:09:19Z Disabled control of system clock
I20260924 04:09:19.722285 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222898807720-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222898807720-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222898807720-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43303
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:45825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43303
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 04:09:19.836653 10104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:09:19.837198 10104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:09:19.837280 10104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:09:19.845003 10104 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:09:19.845099 10104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:09:19.845127 10104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:09:19.845155 10104 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:09:19.845191 10104 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:09:19.853600 10104 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222898807720-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222898807720-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43303
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43303
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10104
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222898807720-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:09:19.855263 10104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:09:19.857448 10104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:09:19.869060 10110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:09:19.869059 10109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:09:19.869127 10112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:09:19.869978 10104 server_base.cc:1034] running on GCE node
I20260924 04:09:19.870693 10104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:09:19.871842 10104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:09:19.873030 10104 hybrid_clock.cc:648] HybridClock initialized: now 1790222959873010 us; error 31 us; skew 500 ppm
I20260924 04:09:19.874904 10104 webserver.cc:460] Webserver started at http://127.0.98.62:34933/ using document root <none> and password file <none>
I20260924 04:09:19.875490 10104 fs_manager.cc:359] Metadata directory not provided
I20260924 04:09:19.875593 10104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:09:19.875839 10104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:09:19.878077 10104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222898807720-392-0/minicluster-data/master-0/data/instance:
uuid: "b2d070a5c7e5464d9a315547aec48135"
format_stamp: "Formatted at 2026-09-24 04:09:19 on dist-test-slave-00ld"
server_key: "7fb550e0039749267313b862fded8d51"
server_key_iv: "b3eb13c1dd16df10db1d5633e3cae1d5"
server_key_version: "encryptionkey@0"
I20260924 04:09:19.878633 10104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222898807720-392-0/minicluster-data/master-0/wal/instance:
uuid: "b2d070a5c7e5464d9a315547aec48135"
format_stamp: "Formatted at 2026-09-24 04:09:19 on dist-test-slave-00ld"
server_key: "7fb550e0039749267313b862fded8d51"
server_key_iv: "b3eb13c1dd16df10db1d5633e3cae1d5"
server_key_version: "encryptionkey@0"
I20260924 04:09:19.882326 10104 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 04:09:19.884790 10118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:09:19.886020 10104 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:09:19.886159 10104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222898807720-392-0/minicluster-data/master-0/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222898807720-392-0/minicluster-data/master-0/wal
uuid: "b2d070a5c7e5464d9a315547aec48135"
format_stamp: "Formatted at 2026-09-24 04:09:19 on dist-test-slave-00ld"
server_key: "7fb550e0039749267313b862fded8d51"
server_key_iv: "b3eb13c1dd16df10db1d5633e3cae1d5"
server_key_version: "encryptionkey@0"
I20260924 04:09:19.886276 10104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222898807720-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222898807720-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222898807720-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:09:19.890740 10104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:09:19.891311 10104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:09:19.891482 10104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:09:19.898332 10104 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43303
I20260924 04:09:19.898346 10170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43303 every 8 connection(s)
I20260924 04:09:19.899196 10104 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:09:19.902181 10171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:09:19.907428 10171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2d070a5c7e5464d9a315547aec48135: Bootstrap starting.
I20260924 04:09:19.907737 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 10104
I20260924 04:09:19.907863 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222898807720-392-0/minicluster-data/master-0/wal/instance
I20260924 04:09:19.908134 392 external_mini_cluster.cc:1428] Setting key 559f7aca29bd630c59399248d7c7a77b
I20260924 04:09:19.909941 10171 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2d070a5c7e5464d9a315547aec48135: Neither blocks nor log segments found. Creating new log.
I20260924 04:09:19.910660 10171 log.cc:824] T 00000000000000000000000000000000 P b2d070a5c7e5464d9a315547aec48135: Log is configured to *not* fsync() on all Append() calls
I20260924 04:09:19.912952 10171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2d070a5c7e5464d9a315547aec48135: No bootstrap required, opened a new log
I20260924 04:09:19.913961 10174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:09:19.909515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38070 (local address 127.0.98.62:43303)
0924 04:09:19.910230 (+ 715us) server_negotiation.cc:206] Beginning negotiation
0924 04:09:19.910246 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 04:09:19.910280 (+ 34us) server_negotiation.cc:407] Connection header received
0924 04:09:19.910947 (+ 667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:09:19.910965 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:09:19.911370 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:09:19.911693 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:09:19.912524 (+ 831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:09:19.912528 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:09:19.912540 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:09:19.912549 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:09:19.912604 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:09:19.912620 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:09:19.912761 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 04:09:19.913034 (+ 273us) server_negotiation.cc:299] Negotiation successful
0924 04:09:19.913294 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":145,"threads_started":1}
I20260924 04:09:19.917193 10171 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2d070a5c7e5464d9a315547aec48135 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2d070a5c7e5464d9a315547aec48135" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43303 } }
I20260924 04:09:19.917488 10171 raft_consensus.cc:374] T 00000000000000000000000000000000 P b2d070a5c7e5464d9a315547aec48135 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:09:19.917547 10171 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2d070a5c7e5464d9a315547aec48135 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2d070a5c7e5464d9a315547aec48135, State: Initialized, Role: FOLLOWER
I20260924 04:09:19.917994 10171 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2d070a5c7e5464d9a315547aec48135 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2d070a5c7e5464d9a315547aec48135" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43303 } }
I20260924 04:09:19.918138 10171 raft_consensus.cc:388] T 00000000000000000000000000000000 P b2d070a5c7e5464d9a315547aec48135 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:09:19.918202 10171 raft_consensus.cc:482] T 00000000000000000000000000000000 P b2d070a5c7e5464d9a315547aec48135 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:09:19.918314 10171 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2d070a5c7e5464d9a315547aec48135 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:09:19.919541 10171 raft_consensus.cc:504] T 00000000000000000000000000000000 P b2d070a5c7e5464d9a315547aec48135 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2d070a5c7e5464d9a315547aec48135" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43303 } }
I20260924 04:09:19.919835 10171 leader_election.cc:302] T 00000000000000000000000000000000 P b2d070a5c7e5464d9a315547aec48135 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b2d070a5c7e5464d9a315547aec48135; no voters:
I20260924 04:09:19.920078 10171 leader_election.cc:288] T 00000000000000000000000000000000 P b2d070a5c7e5464d9a315547aec48135 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:09:19.920220 10176 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b2d070a5c7e5464d9a315547aec48135 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:09:19.920485 10176 raft_consensus.cc:686] T 00000000000000000000000000000000 P b2d070a5c7e5464d9a315547aec48135 [term 1 LEADER]: Becoming Leader. State: Replica: b2d070a5c7e5464d9a315547aec48135, State: Running, Role: LEADER
I20260924 04:09:19.920846 10176 consensus_queue.cc:237] T 00000000000000000000000000000000 P b2d070a5c7e5464d9a315547aec48135 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2d070a5c7e5464d9a315547aec48135" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43303 } }
I20260924 04:09:19.921340 10171 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2d070a5c7e5464d9a315547aec48135 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:09:19.922473 10178 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2d070a5c7e5464d9a315547aec48135 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2d070a5c7e5464d9a315547aec48135. Latest consensus state: current_term: 1 leader_uuid: "b2d070a5c7e5464d9a315547aec48135" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2d070a5c7e5464d9a315547aec48135" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43303 } } }
I20260924 04:09:19.922597 10178 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2d070a5c7e5464d9a315547aec48135 [sys.catalog]: This master's current role is: LEADER
I20260924 04:09:19.922876 10177 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2d070a5c7e5464d9a315547aec48135 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b2d070a5c7e5464d9a315547aec48135" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2d070a5c7e5464d9a315547aec48135" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43303 } } }
I20260924 04:09:19.922950 10185 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:09:19.922976 10177 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2d070a5c7e5464d9a315547aec48135 [sys.catalog]: This master's current role is: LEADER
I20260924 04:09:19.926074 10185 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:09:19.931501 10185 catalog_manager.cc:1347] Generated new cluster ID: aafd9fcdd4c9405d80d7462731ca8f92
I20260924 04:09:19.931568 10185 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:09:19.941911 10185 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:09:19.942238 10185 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:09:19.954149 10185 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b2d070a5c7e5464d9a315547aec48135: Generated new TSK 0
I20260924 04:09:19.954993 10185 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:09:19.962986 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222898807720-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222898807720-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222898807720-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222898807720-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43303
--builtin_ntp_servers=127.0.98.20:45825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 04:09:20.072865 10195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:09:20.073156 10195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:09:20.073237 10195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:09:20.077358 10195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:09:20.077435 10195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:09:20.077521 10195 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:09:20.082199 10195 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222898807720-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222898807720-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222898807720-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10195
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222898807720-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:09:20.083259 10195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:09:20.084653 10195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:09:20.094079 10201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:09:20.094081 10200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:09:20.094069 10203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:09:20.094306 10195 server_base.cc:1034] running on GCE node
I20260924 04:09:20.094691 10195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:09:20.095317 10195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:09:20.096478 10195 hybrid_clock.cc:648] HybridClock initialized: now 1790222960096444 us; error 45 us; skew 500 ppm
I20260924 04:09:20.098310 10195 webserver.cc:460] Webserver started at http://127.0.98.1:36835/ using document root <none> and password file <none>
I20260924 04:09:20.098870 10195 fs_manager.cc:359] Metadata directory not provided
I20260924 04:09:20.098932 10195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:09:20.099193 10195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:09:20.101259 10195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222898807720-392-0/minicluster-data/ts-0/data/instance:
uuid: "3a3b57407f46444e959886520b926c8f"
format_stamp: "Formatted at 2026-09-24 04:09:20 on dist-test-slave-00ld"
server_key: "d1d36ff3009766aa603dfdaa2618794e"
server_key_iv: "e78d2c3367219426effb9d069004f415"
server_key_version: "encryptionkey@0"
I20260924 04:09:20.101814 10195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222898807720-392-0/minicluster-data/ts-0/wal/instance:
uuid: "3a3b57407f46444e959886520b926c8f"
format_stamp: "Formatted at 2026-09-24 04:09:20 on dist-test-slave-00ld"
server_key: "d1d36ff3009766aa603dfdaa2618794e"
server_key_iv: "e78d2c3367219426effb9d069004f415"
server_key_version: "encryptionkey@0"
I20260924 04:09:20.105376 10195 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:09:20.107668 10209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:09:20.108971 10195 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:09:20.109104 10195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222898807720-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222898807720-392-0/minicluster-data/ts-0/wal
uuid: "3a3b57407f46444e959886520b926c8f"
format_stamp: "Formatted at 2026-09-24 04:09:20 on dist-test-slave-00ld"
server_key: "d1d36ff3009766aa603dfdaa2618794e"
server_key_iv: "e78d2c3367219426effb9d069004f415"
server_key_version: "encryptionkey@0"
I20260924 04:09:20.109225 10195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222898807720-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222898807720-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222898807720-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:09:20.113687 10195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:09:20.114257 10195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:09:20.114434 10195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:09:20.114949 10195 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:09:20.115819 10195 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:09:20.115893 10195 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:09:20.115955 10195 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:09:20.116014 10195 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:09:20.125257 10195 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35457
I20260924 04:09:20.125276 10322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35457 every 8 connection(s)
I20260924 04:09:20.126166 10195 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222898807720-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:09:20.129034 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 10195
I20260924 04:09:20.129153 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222898807720-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:09:20.129465 392 external_mini_cluster.cc:1428] Setting key fbf945d92abd4c804a17d7800c325364
I20260924 04:09:20.131771 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222898807720-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222898807720-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222898807720-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222898807720-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43303
--builtin_ntp_servers=127.0.98.20:45825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260924 04:09:20.132864 10174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:09:20.127957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58351 (local address 127.0.98.62:43303)
0924 04:09:20.128105 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 04:09:20.128110 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:09:20.129331 (+ 1221us) server_negotiation.cc:407] Connection header received
0924 04:09:20.131000 (+ 1669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:09:20.131003 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:09:20.131047 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:09:20.131151 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:09:20.132325 (+ 1174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:09:20.132331 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:09:20.132338 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:09:20.132354 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:09:20.132380 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:09:20.132393 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:09:20.132488 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 04:09:20.132609 (+ 121us) server_negotiation.cc:299] Negotiation successful
0924 04:09:20.132656 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 04:09:20.133630 10325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:09:20.128452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43303 (local address 127.0.98.1:58351)
0924 04:09:20.129159 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0924 04:09:20.129212 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 04:09:20.130710 (+ 1498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:09:20.131323 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:09:20.131330 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:09:20.131419 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:09:20.132046 (+ 627us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:09:20.132048 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:09:20.132106 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:09:20.132174 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:09:20.132478 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:09:20.132484 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:09:20.132488 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 04:09:20.132614 (+ 126us) client_negotiation.cc:240] Negotiation successful
0924 04:09:20.132863 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":114,"threads_started":1}
I20260924 04:09:20.134895 10323 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43303
I20260924 04:09:20.135116 10323 heartbeater.cc:464] Registering TS with master...
I20260924 04:09:20.135522 10323 heartbeater.cc:511] Master 127.0.98.62:43303 requested a full tablet report, sending...
I20260924 04:09:20.137069 10135 ts_manager.cc:194] Registered new tserver with Master: 3a3b57407f46444e959886520b926c8f (127.0.98.1:35457)
W20260924 04:09:20.242888 10326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:09:20.243144 10326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:09:20.243181 10326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:09:20.247191 10326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:09:20.247248 10326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:09:20.247339 10326 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:09:20.251986 10326 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222898807720-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222898807720-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222898807720-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10326
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222898807720-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:09:20.253079 10326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:09:20.254431 10326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:09:20.263931 10334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:09:20.263931 10332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:09:20.263940 10331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:09:20.264127 10326 server_base.cc:1034] running on GCE node
I20260924 04:09:20.264599 10326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:09:20.265407 10326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:09:20.266640 10326 hybrid_clock.cc:648] HybridClock initialized: now 1790222960266625 us; error 33 us; skew 500 ppm
I20260924 04:09:20.268518 10326 webserver.cc:460] Webserver started at http://127.0.98.2:42139/ using document root <none> and password file <none>
I20260924 04:09:20.269146 10326 fs_manager.cc:359] Metadata directory not provided
I20260924 04:09:20.269234 10326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:09:20.269469 10326 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:09:20.271570 10326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222898807720-392-0/minicluster-data/ts-1/data/instance:
uuid: "7d8b32ddbc534fbf8fd63868dd168891"
format_stamp: "Formatted at 2026-09-24 04:09:20 on dist-test-slave-00ld"
server_key: "b761dcb3a5f73585fc15114fa802fd77"
server_key_iv: "c6a1ffc23a6c5a043bb41bfc974c136b"
server_key_version: "encryptionkey@0"
I20260924 04:09:20.272143 10326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222898807720-392-0/minicluster-data/ts-1/wal/instance:
uuid: "7d8b32ddbc534fbf8fd63868dd168891"
format_stamp: "Formatted at 2026-09-24 04:09:20 on dist-test-slave-00ld"
server_key: "b761dcb3a5f73585fc15114fa802fd77"
server_key_iv: "c6a1ffc23a6c5a043bb41bfc974c136b"
server_key_version: "encryptionkey@0"
I20260924 04:09:20.275804 10326 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 04:09:20.278247 10340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:09:20.279387 10326 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:09:20.279524 10326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222898807720-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222898807720-392-0/minicluster-data/ts-1/wal
uuid: "7d8b32ddbc534fbf8fd63868dd168891"
format_stamp: "Formatted at 2026-09-24 04:09:20 on dist-test-slave-00ld"
server_key: "b761dcb3a5f73585fc15114fa802fd77"
server_key_iv: "c6a1ffc23a6c5a043bb41bfc974c136b"
server_key_version: "encryptionkey@0"
I20260924 04:09:20.279673 10326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222898807720-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222898807720-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222898807720-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:09:20.284138 10326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:09:20.284775 10326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:09:20.284947 10326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:09:20.285468 10326 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:09:20.286334 10326 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:09:20.286404 10326 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:09:20.286468 10326 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:09:20.286518 10326 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:09:20.295490 10326 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35421
I20260924 04:09:20.295513 10453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35421 every 8 connection(s)
I20260924 04:09:20.296406 10326 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222898807720-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:09:20.298395 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 10326
I20260924 04:09:20.298549 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222898807720-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:09:20.298836 392 external_mini_cluster.cc:1428] Setting key 9d4bf6998fdd1fafd63f3b658228d75d
I20260924 04:09:20.301045 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222898807720-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222898807720-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222898807720-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222898807720-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43303
--builtin_ntp_servers=127.0.98.20:45825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260924 04:09:20.303318 10174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:09:20.298174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57909 (local address 127.0.98.62:43303)
0924 04:09:20.298352 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0924 04:09:20.298357 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:09:20.299599 (+ 1242us) server_negotiation.cc:407] Connection header received
0924 04:09:20.301300 (+ 1701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:09:20.301306 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:09:20.301350 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:09:20.301408 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:09:20.302803 (+ 1395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:09:20.302809 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:09:20.302814 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:09:20.302829 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:09:20.302856 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:09:20.302869 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:09:20.302973 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 04:09:20.303091 (+ 118us) server_negotiation.cc:299] Negotiation successful
0924 04:09:20.303139 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 04:09:20.304270 10456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:09:20.298667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43303 (local address 127.0.98.2:57909)
0924 04:09:20.299437 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0924 04:09:20.299489 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 04:09:20.301052 (+ 1563us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:09:20.301785 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:09:20.301796 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:09:20.301911 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:09:20.302491 (+ 580us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:09:20.302495 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:09:20.302557 (+ 62us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:09:20.302649 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:09:20.302962 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:09:20.302968 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:09:20.302972 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 04:09:20.303102 (+ 130us) client_negotiation.cc:240] Negotiation successful
0924 04:09:20.303290 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":131,"threads_started":1}
I20260924 04:09:20.305475 10454 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43303
I20260924 04:09:20.305737 10454 heartbeater.cc:464] Registering TS with master...
I20260924 04:09:20.306129 10454 heartbeater.cc:511] Master 127.0.98.62:43303 requested a full tablet report, sending...
I20260924 04:09:20.307149 10135 ts_manager.cc:194] Registered new tserver with Master: 7d8b32ddbc534fbf8fd63868dd168891 (127.0.98.2:35421)
W20260924 04:09:20.412820 10457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:09:20.413158 10457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:09:20.413242 10457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:09:20.417335 10457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:09:20.417420 10457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:09:20.417517 10457 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:09:20.422446 10457 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222898807720-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222898807720-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222898807720-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10457
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222898807720-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:09:20.423796 10457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:09:20.425266 10457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:09:20.434962 10465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:09:20.434943 10463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:09:20.434973 10462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:09:20.435377 10457 server_base.cc:1034] running on GCE node
I20260924 04:09:20.435806 10457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:09:20.436503 10457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:09:20.437688 10457 hybrid_clock.cc:648] HybridClock initialized: now 1790222960437655 us; error 43 us; skew 500 ppm
I20260924 04:09:20.439584 10457 webserver.cc:460] Webserver started at http://127.0.98.3:39443/ using document root <none> and password file <none>
I20260924 04:09:20.440199 10457 fs_manager.cc:359] Metadata directory not provided
I20260924 04:09:20.440295 10457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:09:20.440537 10457 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:09:20.442672 10457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222898807720-392-0/minicluster-data/ts-2/data/instance:
uuid: "eadc4c640a5b4020a6ecffcbe5504ada"
format_stamp: "Formatted at 2026-09-24 04:09:20 on dist-test-slave-00ld"
server_key: "1df9d7ff3ecfb588ba31e8ed04b8ea39"
server_key_iv: "50744efff4f33f96cce1559dd135c583"
server_key_version: "encryptionkey@0"
I20260924 04:09:20.443264 10457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222898807720-392-0/minicluster-data/ts-2/wal/instance:
uuid: "eadc4c640a5b4020a6ecffcbe5504ada"
format_stamp: "Formatted at 2026-09-24 04:09:20 on dist-test-slave-00ld"
server_key: "1df9d7ff3ecfb588ba31e8ed04b8ea39"
server_key_iv: "50744efff4f33f96cce1559dd135c583"
server_key_version: "encryptionkey@0"
I20260924 04:09:20.447044 10457 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 04:09:20.449544 10471 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:09:20.450847 10457 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:09:20.450992 10457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222898807720-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222898807720-392-0/minicluster-data/ts-2/wal
uuid: "eadc4c640a5b4020a6ecffcbe5504ada"
format_stamp: "Formatted at 2026-09-24 04:09:20 on dist-test-slave-00ld"
server_key: "1df9d7ff3ecfb588ba31e8ed04b8ea39"
server_key_iv: "50744efff4f33f96cce1559dd135c583"
server_key_version: "encryptionkey@0"
I20260924 04:09:20.451109 10457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222898807720-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222898807720-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222898807720-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:09:20.456008 10457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:09:20.456606 10457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:09:20.456806 10457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:09:20.457351 10457 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:09:20.458271 10457 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:09:20.458341 10457 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:09:20.458410 10457 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:09:20.458462 10457 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:09:20.467378 10457 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38715
I20260924 04:09:20.467396 10584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38715 every 8 connection(s)
I20260924 04:09:20.468299 10457 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222898807720-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:09:20.474150 10174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:09:20.470099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38281 (local address 127.0.98.62:43303)
0924 04:09:20.470243 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 04:09:20.470247 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:09:20.471248 (+ 1001us) server_negotiation.cc:407] Connection header received
0924 04:09:20.472494 (+ 1246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:09:20.472497 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:09:20.472531 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:09:20.472605 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:09:20.473776 (+ 1171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:09:20.473779 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:09:20.473781 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:09:20.473792 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:09:20.473809 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:09:20.473817 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:09:20.473893 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 04:09:20.474003 (+ 110us) server_negotiation.cc:299] Negotiation successful
0924 04:09:20.474038 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 04:09:20.475013 10587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:09:20.470483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43303 (local address 127.0.98.3:38281)
0924 04:09:20.471127 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0924 04:09:20.471166 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 04:09:20.472305 (+ 1139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:09:20.472837 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:09:20.472844 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:09:20.472930 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:09:20.473535 (+ 605us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:09:20.473537 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:09:20.473576 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:09:20.473643 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:09:20.473891 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:09:20.473896 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:09:20.473900 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 04:09:20.474025 (+ 125us) client_negotiation.cc:240] Negotiation successful
0924 04:09:20.474250 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":123,"threads_started":1}
I20260924 04:09:20.476173 10585 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43303
I20260924 04:09:20.476423 10585 heartbeater.cc:464] Registering TS with master...
I20260924 04:09:20.476866 10585 heartbeater.cc:511] Master 127.0.98.62:43303 requested a full tablet report, sending...
I20260924 04:09:20.478017 10135 ts_manager.cc:194] Registered new tserver with Master: eadc4c640a5b4020a6ecffcbe5504ada (127.0.98.3:38715)
I20260924 04:09:20.478171 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 10457
I20260924 04:09:20.478281 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790222898807720-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:09:20.478588 392 external_mini_cluster.cc:1428] Setting key 37d3fdd514e59fa2901bc2c72e92c013
I20260924 04:09:20.480568 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 04:09:20.483712 10174 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:09:20.482629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38084 (local address 127.0.98.62:43303)
0924 04:09:20.482765 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 04:09:20.482768 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:09:20.482996 (+ 228us) server_negotiation.cc:407] Connection header received
0924 04:09:20.483064 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:09:20.483066 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:09:20.483100 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:09:20.483185 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:09:20.483512 (+ 327us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38084: BlockingRecv error: recv got EOF from 127.0.0.1:38084 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 04:09:20.483773 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 10195
I20260924 04:09:20.489547 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 10326
I20260924 04:09:20.494578 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 10457
I20260924 04:09:20.499790 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 10104
2026-09-24T04:09:20Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (806 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222898807720-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:09:20 dist-test-slave-00ld krb5kdc[10595](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:09:20 dist-test-slave-00ld krb5kdc[10595](info): set up 2 sockets
Sep 24 04:09:20 dist-test-slave-00ld krb5kdc[10595](info): commencing operation
krb5kdc: starting...
W20260924 04:09:21.546092 392 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222898807720-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222898807720-392-0/krb5kdc/test-user.keytab.
Sep 24 04:09:21 dist-test-slave-00ld krb5kdc[10595](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222961, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:09:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:09:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:09:21.668884 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222898807720-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222898807720-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222898807720-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35965
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35965
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:09:21.776398 10608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:09:21.776662 10608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:09:21.776783 10608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:09:21.780941 10608 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:09:21.781005 10608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:09:21.781024 10608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:09:21.781056 10608 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:09:21.781093 10608 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:09:21.786379 10608 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222898807720-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222898807720-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35965
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35965
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10608
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222898807720-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:09:21.787443 10608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:09:21.788456 10608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:09:21.796720 10613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:09:21.796763 10614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:09:21.796878 10608 server_base.cc:1034] running on GCE node
W20260924 04:09:21.797055 10616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:09:21.797626 10608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:09:21.798623 10608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:09:21.799788 10608 hybrid_clock.cc:648] HybridClock initialized: now 1790222961799757 us; error 44 us; skew 500 ppm
Sep 24 04:09:21 dist-test-slave-00ld krb5kdc[10595](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222961, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:09:21.802464 10608 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:09:21.803363 10608 webserver.cc:460] Webserver started at http://127.0.98.62:33317/ using document root <none> and password file <none>
I20260924 04:09:21.803910 10608 fs_manager.cc:359] Metadata directory not provided
I20260924 04:09:21.803996 10608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:09:21.804224 10608 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:09:21.806285 10608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222898807720-392-0/minicluster-data/master-0/data/instance:
uuid: "24a6017fc4344885a8a392ceb9496eac"
format_stamp: "Formatted at 2026-09-24 04:09:21 on dist-test-slave-00ld"
server_key: "42683813162c4ec8d8a99d2523376373"
server_key_iv: "0f90f18dc827e24d41a1e4d76043f7f7"
server_key_version: "encryptionkey@0"
I20260924 04:09:21.806840 10608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222898807720-392-0/minicluster-data/master-0/wal/instance:
uuid: "24a6017fc4344885a8a392ceb9496eac"
format_stamp: "Formatted at 2026-09-24 04:09:21 on dist-test-slave-00ld"
server_key: "42683813162c4ec8d8a99d2523376373"
server_key_iv: "0f90f18dc827e24d41a1e4d76043f7f7"
server_key_version: "encryptionkey@0"
I20260924 04:09:21.810487 10608 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260924 04:09:21.812949 10623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:09:21.814204 10608 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:09:21.814342 10608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222898807720-392-0/minicluster-data/master-0/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222898807720-392-0/minicluster-data/master-0/wal
uuid: "24a6017fc4344885a8a392ceb9496eac"
format_stamp: "Formatted at 2026-09-24 04:09:21 on dist-test-slave-00ld"
server_key: "42683813162c4ec8d8a99d2523376373"
server_key_iv: "0f90f18dc827e24d41a1e4d76043f7f7"
server_key_version: "encryptionkey@0"
I20260924 04:09:21.814452 10608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222898807720-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222898807720-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222898807720-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:09:21.829747 10608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:09:21.832772 10608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:09:21.832966 10608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:09:21.839978 10608 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35965
I20260924 04:09:21.840303 10675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35965 every 8 connection(s)
I20260924 04:09:21.841037 10608 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:09:21.843779 10676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:09:21.845322 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 10608
I20260924 04:09:21.845443 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222898807720-392-0/minicluster-data/master-0/wal/instance
I20260924 04:09:21.845742 392 external_mini_cluster.cc:1428] Setting key 684212393c0664e2f283b70f091d4959
I20260924 04:09:21.849936 10676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24a6017fc4344885a8a392ceb9496eac: Bootstrap starting.
I20260924 04:09:21.852066 10676 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 24a6017fc4344885a8a392ceb9496eac: Neither blocks nor log segments found. Creating new log.
I20260924 04:09:21.852941 10676 log.cc:824] T 00000000000000000000000000000000 P 24a6017fc4344885a8a392ceb9496eac: Log is configured to *not* fsync() on all Append() calls
Sep 24 04:09:21 dist-test-slave-00ld krb5kdc[10595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222961, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:09:21.855273 10676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24a6017fc4344885a8a392ceb9496eac: No bootstrap required, opened a new log
I20260924 04:09:21.859475 10676 raft_consensus.cc:348] T 00000000000000000000000000000000 P 24a6017fc4344885a8a392ceb9496eac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24a6017fc4344885a8a392ceb9496eac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35965 } }
I20260924 04:09:21.859690 10676 raft_consensus.cc:374] T 00000000000000000000000000000000 P 24a6017fc4344885a8a392ceb9496eac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:09:21.859737 10679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:09:21.847200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44638 (local address 127.0.98.62:35965)
0924 04:09:21.847617 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0924 04:09:21.847632 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 04:09:21.847665 (+ 33us) server_negotiation.cc:407] Connection header received
0924 04:09:21.848281 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:09:21.848296 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:09:21.848601 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:09:21.848978 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:09:21.849754 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:21.850733 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:21.851549 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:21.852006 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:21.852166 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:21.855163 (+ 2997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:09:21.855167 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:09:21.855180 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:09:21.855206 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:09:21.856943 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:09:21.857469 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:09:21.857473 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:09:21.857477 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:09:21.857576 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:09:21.857856 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:09:21.857858 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:09:21.857859 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:09:21.858219 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:09:21.858366 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 04:09:21.858656 (+ 290us) server_negotiation.cc:299] Negotiation successful
0924 04:09:21.858957 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":132,"threads_started":1}
I20260924 04:09:21.859750 10676 raft_consensus.cc:729] T 00000000000000000000000000000000 P 24a6017fc4344885a8a392ceb9496eac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24a6017fc4344885a8a392ceb9496eac, State: Initialized, Role: FOLLOWER
I20260924 04:09:21.860220 10676 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24a6017fc4344885a8a392ceb9496eac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24a6017fc4344885a8a392ceb9496eac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35965 } }
I20260924 04:09:21.860376 10676 raft_consensus.cc:388] T 00000000000000000000000000000000 P 24a6017fc4344885a8a392ceb9496eac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:09:21.860462 10676 raft_consensus.cc:482] T 00000000000000000000000000000000 P 24a6017fc4344885a8a392ceb9496eac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:09:21.860584 10676 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 24a6017fc4344885a8a392ceb9496eac [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:09:21.861495 10676 raft_consensus.cc:504] T 00000000000000000000000000000000 P 24a6017fc4344885a8a392ceb9496eac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24a6017fc4344885a8a392ceb9496eac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35965 } }
I20260924 04:09:21.861845 10676 leader_election.cc:302] T 00000000000000000000000000000000 P 24a6017fc4344885a8a392ceb9496eac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 24a6017fc4344885a8a392ceb9496eac; no voters:
I20260924 04:09:21.862164 10676 leader_election.cc:288] T 00000000000000000000000000000000 P 24a6017fc4344885a8a392ceb9496eac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:09:21.862265 10681 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 24a6017fc4344885a8a392ceb9496eac [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:09:21.862475 10681 raft_consensus.cc:686] T 00000000000000000000000000000000 P 24a6017fc4344885a8a392ceb9496eac [term 1 LEADER]: Becoming Leader. State: Replica: 24a6017fc4344885a8a392ceb9496eac, State: Running, Role: LEADER
I20260924 04:09:21.862774 10681 consensus_queue.cc:237] T 00000000000000000000000000000000 P 24a6017fc4344885a8a392ceb9496eac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24a6017fc4344885a8a392ceb9496eac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35965 } }
I20260924 04:09:21.863402 10676 sys_catalog.cc:564] T 00000000000000000000000000000000 P 24a6017fc4344885a8a392ceb9496eac [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:09:21.864449 10683 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24a6017fc4344885a8a392ceb9496eac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24a6017fc4344885a8a392ceb9496eac. Latest consensus state: current_term: 1 leader_uuid: "24a6017fc4344885a8a392ceb9496eac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24a6017fc4344885a8a392ceb9496eac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35965 } } }
I20260924 04:09:21.864586 10683 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24a6017fc4344885a8a392ceb9496eac [sys.catalog]: This master's current role is: LEADER
I20260924 04:09:21.864907 10682 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24a6017fc4344885a8a392ceb9496eac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "24a6017fc4344885a8a392ceb9496eac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24a6017fc4344885a8a392ceb9496eac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35965 } } }
I20260924 04:09:21.864998 10682 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24a6017fc4344885a8a392ceb9496eac [sys.catalog]: This master's current role is: LEADER
I20260924 04:09:21.865283 10690 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:09:21.867923 10690 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:09:21.873263 10690 catalog_manager.cc:1347] Generated new cluster ID: 1112128dfef9447dba87066db8c373b3
I20260924 04:09:21.873340 10690 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:09:21.890574 10690 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:09:21.891587 10690 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:09:21.901168 10690 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 24a6017fc4344885a8a392ceb9496eac: Generated new TSK 0
I20260924 04:09:21.901923 10690 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222898807720-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222898807720-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:09:21.931211 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222898807720-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222898807720-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222898807720-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222898807720-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35965
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222898807720-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:09:22.040508 10702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:09:22.040773 10702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:09:22.040814 10702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:09:22.044694 10702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:09:22.044749 10702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:09:22.044802 10702 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:09:22.049901 10702 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222898807720-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222898807720-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222898807720-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222898807720-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35965
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10702
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222898807720-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:09:22.050877 10702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:09:22.051819 10702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:09:22.061362 10708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:09:22.061296 10707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:09:22.061473 10710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:09:22.061388 10702 server_base.cc:1034] running on GCE node
I20260924 04:09:22.062023 10702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:09:22.062638 10702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:09:22.063844 10702 hybrid_clock.cc:648] HybridClock initialized: now 1790222962063823 us; error 44 us; skew 500 ppm
Sep 24 04:09:22 dist-test-slave-00ld krb5kdc[10595](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222962, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:09:22.066614 10702 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:09:22.067435 10702 webserver.cc:460] Webserver started at http://127.0.98.1:40907/ using document root <none> and password file <none>
I20260924 04:09:22.067991 10702 fs_manager.cc:359] Metadata directory not provided
I20260924 04:09:22.068074 10702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:09:22.068284 10702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:09:22.070312 10702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222898807720-392-0/minicluster-data/ts-0/data/instance:
uuid: "35a9659542c04f3a85eef35cd6488d4c"
format_stamp: "Formatted at 2026-09-24 04:09:22 on dist-test-slave-00ld"
server_key: "3a7be9c725a4503a673319ddada66606"
server_key_iv: "93828a5672644e964527909de1ccd305"
server_key_version: "encryptionkey@0"
I20260924 04:09:22.070876 10702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222898807720-392-0/minicluster-data/ts-0/wal/instance:
uuid: "35a9659542c04f3a85eef35cd6488d4c"
format_stamp: "Formatted at 2026-09-24 04:09:22 on dist-test-slave-00ld"
server_key: "3a7be9c725a4503a673319ddada66606"
server_key_iv: "93828a5672644e964527909de1ccd305"
server_key_version: "encryptionkey@0"
I20260924 04:09:22.074440 10702 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260924 04:09:22.076778 10717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:09:22.077957 10702 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:09:22.078086 10702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222898807720-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222898807720-392-0/minicluster-data/ts-0/wal
uuid: "35a9659542c04f3a85eef35cd6488d4c"
format_stamp: "Formatted at 2026-09-24 04:09:22 on dist-test-slave-00ld"
server_key: "3a7be9c725a4503a673319ddada66606"
server_key_iv: "93828a5672644e964527909de1ccd305"
server_key_version: "encryptionkey@0"
I20260924 04:09:22.078186 10702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222898807720-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222898807720-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222898807720-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:09:22.098285 10702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:09:22.101253 10702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:09:22.101454 10702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:09:22.102035 10702 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:09:22.102830 10702 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:09:22.102900 10702 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:09:22.102969 10702 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:09:22.103019 10702 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:09:22.111796 10702 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45595
I20260924 04:09:22.111819 10830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45595 every 8 connection(s)
I20260924 04:09:22.112605 10702 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222898807720-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:09:22.117298 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 10702
I20260924 04:09:22.117434 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222898807720-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:09:22.117741 392 external_mini_cluster.cc:1428] Setting key 1051c3ed0f8e7a104d1933f7878c4c2c
Sep 24 04:09:22 dist-test-slave-00ld krb5kdc[10595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222962, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:09:22.126248 10679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:09:22.114306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59153 (local address 127.0.98.62:35965)
0924 04:09:22.114446 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 04:09:22.114450 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:09:22.115393 (+ 943us) server_negotiation.cc:407] Connection header received
0924 04:09:22.116238 (+ 845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:09:22.116240 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:09:22.116286 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:09:22.116354 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:09:22.117700 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:22.118441 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:22.119544 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:22.119793 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:22.119901 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:22.122358 (+ 2457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:09:22.122364 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:09:22.122371 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:09:22.122402 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:09:22.124160 (+ 1758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:09:22.124648 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:09:22.124654 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:09:22.124657 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:09:22.124786 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:09:22.125104 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:09:22.125109 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:09:22.125113 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:09:22.125338 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:09:22.125444 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 04:09:22.125902 (+ 458us) server_negotiation.cc:299] Negotiation successful
0924 04:09:22.126047 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 04:09:22.126935 10833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:09:22.114651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35965 (local address 127.0.98.1:59153)
0924 04:09:22.115275 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0924 04:09:22.115313 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 04:09:22.116041 (+ 728us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:09:22.116512 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:09:22.116518 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:09:22.116936 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:09:22.117546 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:22.117562 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:22.118556 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:22.118560 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:22.119422 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:22.119431 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:22.119933 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:22.119937 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:22.120114 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:22.120773 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:09:22.120789 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:09:22.122198 (+ 1409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:09:22.124288 (+ 2090us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:09:22.124293 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:09:22.124304 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:09:22.124549 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:09:22.124876 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:09:22.124878 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:09:22.124879 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:09:22.125006 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:09:22.125439 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:09:22.125444 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:09:22.125668 (+ 224us) client_negotiation.cc:769] Sending connection context
0924 04:09:22.125823 (+ 155us) client_negotiation.cc:240] Negotiation successful
0924 04:09:22.126058 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":105,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 04:09:22.128171 10831 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35965
I20260924 04:09:22.128432 10831 heartbeater.cc:464] Registering TS with master...
I20260924 04:09:22.128998 10831 heartbeater.cc:511] Master 127.0.98.62:35965 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 04:09:22.130802 10640 ts_manager.cc:194] Registered new tserver with Master: 35a9659542c04f3a85eef35cd6488d4c (127.0.98.1:45595)
I20260924 04:09:22.132105 10640 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:59153
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222898807720-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222898807720-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:09:22.145334 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222898807720-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222898807720-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222898807720-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222898807720-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35965
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222898807720-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:09:22.264228 10836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:09:22.264586 10836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:09:22.264637 10836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:09:22.269264 10836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:09:22.269335 10836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:09:22.269402 10836 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:09:22.275357 10836 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222898807720-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222898807720-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222898807720-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222898807720-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35965
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10836
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222898807720-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:09:22.276489 10836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:09:22.277546 10836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:09:22.287454 10842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:09:22.287452 10841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:09:22.287493 10844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:09:22.287689 10836 server_base.cc:1034] running on GCE node
I20260924 04:09:22.288283 10836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:09:22.288990 10836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:09:22.290176 10836 hybrid_clock.cc:648] HybridClock initialized: now 1790222962290142 us; error 47 us; skew 500 ppm
Sep 24 04:09:22 dist-test-slave-00ld krb5kdc[10595](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222962, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:09:22.292919 10836 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:09:22.293754 10836 webserver.cc:460] Webserver started at http://127.0.98.2:34845/ using document root <none> and password file <none>
I20260924 04:09:22.294306 10836 fs_manager.cc:359] Metadata directory not provided
I20260924 04:09:22.294397 10836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:09:22.294622 10836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:09:22.296617 10836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222898807720-392-0/minicluster-data/ts-1/data/instance:
uuid: "95efe778539a4c61b9e39e1d058ecc48"
format_stamp: "Formatted at 2026-09-24 04:09:22 on dist-test-slave-00ld"
server_key: "a86e75394429ee24bd1b9c09436000df"
server_key_iv: "308ee6748ef035af7da6a2062c660d08"
server_key_version: "encryptionkey@0"
I20260924 04:09:22.297227 10836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222898807720-392-0/minicluster-data/ts-1/wal/instance:
uuid: "95efe778539a4c61b9e39e1d058ecc48"
format_stamp: "Formatted at 2026-09-24 04:09:22 on dist-test-slave-00ld"
server_key: "a86e75394429ee24bd1b9c09436000df"
server_key_iv: "308ee6748ef035af7da6a2062c660d08"
server_key_version: "encryptionkey@0"
I20260924 04:09:22.300720 10836 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260924 04:09:22.302971 10851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:09:22.304162 10836 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:09:22.304291 10836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222898807720-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222898807720-392-0/minicluster-data/ts-1/wal
uuid: "95efe778539a4c61b9e39e1d058ecc48"
format_stamp: "Formatted at 2026-09-24 04:09:22 on dist-test-slave-00ld"
server_key: "a86e75394429ee24bd1b9c09436000df"
server_key_iv: "308ee6748ef035af7da6a2062c660d08"
server_key_version: "encryptionkey@0"
I20260924 04:09:22.304394 10836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222898807720-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222898807720-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222898807720-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:09:22.316347 10836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:09:22.319500 10836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:09:22.319697 10836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:09:22.320294 10836 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:09:22.321179 10836 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:09:22.321246 10836 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:09:22.321311 10836 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:09:22.321358 10836 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:09:22.330173 10836 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35495
I20260924 04:09:22.330189 10964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35495 every 8 connection(s)
I20260924 04:09:22.331199 10836 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222898807720-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:09:22.331694 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 10836
I20260924 04:09:22.331822 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222898807720-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:09:22.332166 392 external_mini_cluster.cc:1428] Setting key 82445f136e03c40e9731b623694a2af5
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Sep 24 04:09:22 dist-test-slave-00ld krb5kdc[10595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222962, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260924 04:09:22.346805 10679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:09:22.333228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35331 (local address 127.0.98.62:35965)
0924 04:09:22.333368 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 04:09:22.333373 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:09:22.334649 (+ 1276us) server_negotiation.cc:407] Connection header received
0924 04:09:22.335960 (+ 1311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:09:22.335965 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:09:22.336057 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:09:22.336176 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:09:22.337503 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:22.338467 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:22.339645 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:22.340029 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:22.340155 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:22.342923 (+ 2768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:09:22.342925 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:09:22.342927 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:09:22.342951 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:09:22.344399 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:09:22.344962 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:09:22.344969 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:09:22.344974 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:09:22.345067 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:09:22.345495 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:09:22.345497 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:09:22.345498 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:09:22.345685 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:09:22.345762 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 04:09:22.346425 (+ 663us) server_negotiation.cc:299] Negotiation successful
0924 04:09:22.346588 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 04:09:22.347636 10967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:09:22.333691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35965 (local address 127.0.98.2:35331)
0924 04:09:22.334463 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0924 04:09:22.334534 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0924 04:09:22.335749 (+ 1215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:09:22.336328 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:09:22.336334 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:09:22.336833 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:09:22.337354 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:22.337368 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:22.338643 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:22.338646 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:22.339522 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:22.339530 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:22.340197 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:22.340201 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:22.340348 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:22.340994 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:09:22.341012 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:09:22.342764 (+ 1752us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:09:22.344501 (+ 1737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:09:22.344505 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:09:22.344517 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:09:22.344864 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:09:22.345185 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:09:22.345187 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:09:22.345189 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:09:22.345381 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:09:22.345793 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:09:22.345803 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:09:22.346154 (+ 351us) client_negotiation.cc:769] Sending connection context
0924 04:09:22.346374 (+ 220us) client_negotiation.cc:240] Negotiation successful
0924 04:09:22.346746 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":114,"threads_started":1}
I20260924 04:09:22.348950 10965 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35965
I20260924 04:09:22.349242 10965 heartbeater.cc:464] Registering TS with master...
I20260924 04:09:22.349867 10965 heartbeater.cc:511] Master 127.0.98.62:35965 requested a full tablet report, sending...
I20260924 04:09:22.351054 10640 ts_manager.cc:194] Registered new tserver with Master: 95efe778539a4c61b9e39e1d058ecc48 (127.0.98.2:35495)
I20260924 04:09:22.351639 10640 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35331
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222898807720-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222898807720-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:09:22.361953 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222898807720-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222898807720-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222898807720-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222898807720-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35965
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222898807720-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:09:22.472352 10970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:09:22.472659 10970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:09:22.472756 10970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:09:22.476786 10970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:09:22.476866 10970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:09:22.476958 10970 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:09:22.482267 10970 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222898807720-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222898807720-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222898807720-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222898807720-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35965
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10970
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222898807720-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:09:22.483325 10970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:09:22.484382 10970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:09:22.493741 10975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:09:22.493784 10978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:09:22.493784 10976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:09:22.493950 10970 server_base.cc:1034] running on GCE node
I20260924 04:09:22.494405 10970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:09:22.494997 10970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:09:22.496176 10970 hybrid_clock.cc:648] HybridClock initialized: now 1790222962496149 us; error 49 us; skew 500 ppm
Sep 24 04:09:22 dist-test-slave-00ld krb5kdc[10595](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222962, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:09:22.498699 10970 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:09:22.499550 10970 webserver.cc:460] Webserver started at http://127.0.98.3:44119/ using document root <none> and password file <none>
I20260924 04:09:22.500097 10970 fs_manager.cc:359] Metadata directory not provided
I20260924 04:09:22.500185 10970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:09:22.500407 10970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:09:22.502395 10970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222898807720-392-0/minicluster-data/ts-2/data/instance:
uuid: "c0c47ee86ebe4ea78d331fd0982bb610"
format_stamp: "Formatted at 2026-09-24 04:09:22 on dist-test-slave-00ld"
server_key: "2235665f4d3d201bfd9e7442836fe23c"
server_key_iv: "90ecd0359b5098b907bef80d646a1776"
server_key_version: "encryptionkey@0"
I20260924 04:09:22.502938 10970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222898807720-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c0c47ee86ebe4ea78d331fd0982bb610"
format_stamp: "Formatted at 2026-09-24 04:09:22 on dist-test-slave-00ld"
server_key: "2235665f4d3d201bfd9e7442836fe23c"
server_key_iv: "90ecd0359b5098b907bef80d646a1776"
server_key_version: "encryptionkey@0"
I20260924 04:09:22.506642 10970 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 04:09:22.509238 10985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:09:22.510430 10970 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:09:22.510572 10970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222898807720-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222898807720-392-0/minicluster-data/ts-2/wal
uuid: "c0c47ee86ebe4ea78d331fd0982bb610"
format_stamp: "Formatted at 2026-09-24 04:09:22 on dist-test-slave-00ld"
server_key: "2235665f4d3d201bfd9e7442836fe23c"
server_key_iv: "90ecd0359b5098b907bef80d646a1776"
server_key_version: "encryptionkey@0"
I20260924 04:09:22.510700 10970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222898807720-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222898807720-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222898807720-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:09:22.538702 10970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:09:22.541718 10970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:09:22.541926 10970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:09:22.542522 10970 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:09:22.543372 10970 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:09:22.543445 10970 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:09:22.543517 10970 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:09:22.543560 10970 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:09:22.552201 10970 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34013
I20260924 04:09:22.552229 11098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34013 every 8 connection(s)
I20260924 04:09:22.553157 10970 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222898807720-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:09:22.558638 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 10970
I20260924 04:09:22.558787 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222898807720-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:09:22.559079 392 external_mini_cluster.cc:1428] Setting key 081f4c7567170a31d7b45e68a945c816
Sep 24 04:09:22 dist-test-slave-00ld krb5kdc[10595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222962, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:09:22.566644 10679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:09:22.554856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59219 (local address 127.0.98.62:35965)
0924 04:09:22.555010 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 04:09:22.555014 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:09:22.556103 (+ 1089us) server_negotiation.cc:407] Connection header received
0924 04:09:22.557037 (+ 934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:09:22.557040 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:09:22.557086 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:09:22.557170 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:09:22.558412 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:22.559200 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:22.560591 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:22.560911 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:22.561017 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:22.563424 (+ 2407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:09:22.563427 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:09:22.563429 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:09:22.563451 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:09:22.564900 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:09:22.565431 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:09:22.565434 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:09:22.565435 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:09:22.565507 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:09:22.565766 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:09:22.565768 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:09:22.565770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:09:22.565930 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:09:22.566001 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 04:09:22.566440 (+ 439us) server_negotiation.cc:299] Negotiation successful
0924 04:09:22.566526 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 04:09:22.567468 11101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:09:22.555301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35965 (local address 127.0.98.3:59219)
0924 04:09:22.555947 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0924 04:09:22.555987 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 04:09:22.556846 (+ 859us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:09:22.557367 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:09:22.557374 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:09:22.557766 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:09:22.558257 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:22.558270 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:22.559323 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:22.559326 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:22.560491 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:22.560500 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:22.561088 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:22.561093 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:22.561293 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:22.561847 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:09:22.561863 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:09:22.563274 (+ 1411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:09:22.565022 (+ 1748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:09:22.565027 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:09:22.565039 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:09:22.565346 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:09:22.565566 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:09:22.565568 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:09:22.565569 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:09:22.565693 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:09:22.566012 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:09:22.566017 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:09:22.566248 (+ 231us) client_negotiation.cc:769] Sending connection context
0924 04:09:22.566401 (+ 153us) client_negotiation.cc:240] Negotiation successful
0924 04:09:22.566646 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":121,"threads_started":1}
I20260924 04:09:22.568589 11099 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35965
I20260924 04:09:22.568851 11099 heartbeater.cc:464] Registering TS with master...
I20260924 04:09:22.569316 11099 heartbeater.cc:511] Master 127.0.98.62:35965 requested a full tablet report, sending...
I20260924 04:09:22.570274 10640 ts_manager.cc:194] Registered new tserver with Master: c0c47ee86ebe4ea78d331fd0982bb610 (127.0.98.3:34013)
I20260924 04:09:22.570791 10640 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59219
I20260924 04:09:22.572899 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:09:22.582049 10679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:09:22.574702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44648 (local address 127.0.98.62:35965)
0924 04:09:22.574849 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 04:09:22.574853 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:09:22.575019 (+ 166us) server_negotiation.cc:407] Connection header received
0924 04:09:22.575177 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:09:22.575181 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:09:22.575226 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:09:22.575307 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:09:22.575988 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:22.576573 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:22.577442 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:22.577659 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:22.577769 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:22.579094 (+ 1325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:09:22.579098 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:09:22.579101 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:09:22.579132 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:09:22.580298 (+ 1166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:09:22.580763 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:09:22.580767 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:09:22.580769 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:09:22.580848 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:09:22.581129 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:09:22.581134 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:09:22.581135 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:09:22.581334 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:09:22.581475 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 04:09:22.581819 (+ 344us) server_negotiation.cc:299] Negotiation successful
0924 04:09:22.581925 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 04:09:22.584553 10640 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44648:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:09:22.587419 10640 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:09:22.601799 11110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:09:22.595374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34013 (local address 127.0.0.1:58734)
0924 04:09:22.595747 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0924 04:09:22.595795 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 04:09:22.595944 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:09:22.596796 (+ 852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:09:22.596801 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:09:22.596825 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:09:22.596996 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:22.597003 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:22.598672 (+ 1669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:22.598675 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:22.600329 (+ 1654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:22.600340 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:22.601266 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:22.601270 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:22.601375 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:22.601410 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 04:09:22.601526 (+ 116us) client_negotiation.cc:240] Negotiation successful
0924 04:09:22.601650 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":175,"thread_start_us":85,"threads_started":1}
I20260924 04:09:22.601807 11111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:09:22.595524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58734 (local address 127.0.98.3:34013)
0924 04:09:22.596241 (+ 717us) server_negotiation.cc:206] Beginning negotiation
0924 04:09:22.596254 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 04:09:22.596277 (+ 23us) server_negotiation.cc:407] Connection header received
0924 04:09:22.596356 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:09:22.596362 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:09:22.596580 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:09:22.597166 (+ 586us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:09:22.597335 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:22.598507 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:22.600477 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:22.601138 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:22.601266 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:22.601417 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 04:09:22.601562 (+ 145us) server_negotiation.cc:299] Negotiation successful
0924 04:09:22.601655 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":147,"threads_started":1}
I20260924 04:09:22.602149 11115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:09:22.596571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58716 (local address 127.0.98.1:45595)
0924 04:09:22.596841 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0924 04:09:22.596855 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 04:09:22.597010 (+ 155us) server_negotiation.cc:407] Connection header received
0924 04:09:22.597153 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:09:22.597158 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:09:22.597403 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:09:22.597545 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:09:22.597837 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:22.599050 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:22.600296 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:22.601177 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:22.601291 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:22.601437 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0924 04:09:22.601892 (+ 455us) server_negotiation.cc:299] Negotiation successful
0924 04:09:22.601987 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":62,"threads_started":1}
I20260924 04:09:22.602310 11113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:09:22.596149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45595 (local address 127.0.0.1:58716)
0924 04:09:22.596908 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0924 04:09:22.596921 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:09:22.597010 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:09:22.597605 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:09:22.597610 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:09:22.597631 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:09:22.597722 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:22.597729 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:22.599204 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:22.599207 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:22.600172 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:22.600181 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:22.601502 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:22.601506 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:22.601675 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:22.601738 (+ 63us) client_negotiation.cc:769] Sending connection context
0924 04:09:22.601807 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 04:09:22.602154 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":698,"mutex_wait_us":357,"thread_start_us":78,"threads_started":1}
I20260924 04:09:22.602732 11114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:09:22.596272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59686 (local address 127.0.98.2:35495)
0924 04:09:22.597031 (+ 759us) server_negotiation.cc:206] Beginning negotiation
0924 04:09:22.597049 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 04:09:22.597072 (+ 23us) server_negotiation.cc:407] Connection header received
0924 04:09:22.597156 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:09:22.597160 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:09:22.597292 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:09:22.597442 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:09:22.597684 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:22.598445 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:22.600451 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:22.601255 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:22.601340 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:22.601463 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 04:09:22.602491 (+ 1028us) server_negotiation.cc:299] Negotiation successful
0924 04:09:22.602602 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":634,"thread_start_us":127,"threads_started":1}
I20260924 04:09:22.602962 11112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:09:22.595591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35495 (local address 127.0.0.1:59686)
0924 04:09:22.596650 (+ 1059us) negotiation.cc:107] Waiting for socket to connect
0924 04:09:22.596668 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 04:09:22.596776 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:09:22.597421 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:09:22.597424 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:09:22.597440 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:09:22.597556 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:22.597566 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:22.598582 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:22.598588 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:22.600329 (+ 1741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:22.600340 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:22.601509 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:22.601512 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:22.601663 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:22.601737 (+ 74us) client_negotiation.cc:769] Sending connection context
0924 04:09:22.601807 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 04:09:22.602860 (+ 1053us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":990,"mutex_wait_us":301,"thread_start_us":91,"threads_started":1}
I20260924 04:09:22.605538 10765 tablet_service.cc:1467] Processing CreateTablet for tablet bf22324b81b847d2b593830e1d070bc5 (DEFAULT_TABLE table=test-table [id=92a98efb912a4026bf565c534b1af187]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:09:22.605530 10899 tablet_service.cc:1467] Processing CreateTablet for tablet bf22324b81b847d2b593830e1d070bc5 (DEFAULT_TABLE table=test-table [id=92a98efb912a4026bf565c534b1af187]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:09:22.605885 11033 tablet_service.cc:1467] Processing CreateTablet for tablet bf22324b81b847d2b593830e1d070bc5 (DEFAULT_TABLE table=test-table [id=92a98efb912a4026bf565c534b1af187]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:09:22.606675 10765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf22324b81b847d2b593830e1d070bc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:09:22.606674 10899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf22324b81b847d2b593830e1d070bc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:09:22.606690 11033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf22324b81b847d2b593830e1d070bc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:09:22.612895 11116 tablet_bootstrap.cc:492] T bf22324b81b847d2b593830e1d070bc5 P 95efe778539a4c61b9e39e1d058ecc48: Bootstrap starting.
I20260924 04:09:22.614266 11117 tablet_bootstrap.cc:492] T bf22324b81b847d2b593830e1d070bc5 P c0c47ee86ebe4ea78d331fd0982bb610: Bootstrap starting.
I20260924 04:09:22.615939 11116 tablet_bootstrap.cc:654] T bf22324b81b847d2b593830e1d070bc5 P 95efe778539a4c61b9e39e1d058ecc48: Neither blocks nor log segments found. Creating new log.
I20260924 04:09:22.616384 11117 tablet_bootstrap.cc:654] T bf22324b81b847d2b593830e1d070bc5 P c0c47ee86ebe4ea78d331fd0982bb610: Neither blocks nor log segments found. Creating new log.
I20260924 04:09:22.616972 11118 tablet_bootstrap.cc:492] T bf22324b81b847d2b593830e1d070bc5 P 35a9659542c04f3a85eef35cd6488d4c: Bootstrap starting.
I20260924 04:09:22.617096 11116 log.cc:824] T bf22324b81b847d2b593830e1d070bc5 P 95efe778539a4c61b9e39e1d058ecc48: Log is configured to *not* fsync() on all Append() calls
I20260924 04:09:22.617586 11117 log.cc:824] T bf22324b81b847d2b593830e1d070bc5 P c0c47ee86ebe4ea78d331fd0982bb610: Log is configured to *not* fsync() on all Append() calls
I20260924 04:09:22.619086 11118 tablet_bootstrap.cc:654] T bf22324b81b847d2b593830e1d070bc5 P 35a9659542c04f3a85eef35cd6488d4c: Neither blocks nor log segments found. Creating new log.
I20260924 04:09:22.619215 11116 tablet_bootstrap.cc:492] T bf22324b81b847d2b593830e1d070bc5 P 95efe778539a4c61b9e39e1d058ecc48: No bootstrap required, opened a new log
I20260924 04:09:22.619261 11117 tablet_bootstrap.cc:492] T bf22324b81b847d2b593830e1d070bc5 P c0c47ee86ebe4ea78d331fd0982bb610: No bootstrap required, opened a new log
I20260924 04:09:22.619390 11117 ts_tablet_manager.cc:1397] T bf22324b81b847d2b593830e1d070bc5 P c0c47ee86ebe4ea78d331fd0982bb610: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 04:09:22.619403 11116 ts_tablet_manager.cc:1397] T bf22324b81b847d2b593830e1d070bc5 P 95efe778539a4c61b9e39e1d058ecc48: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20260924 04:09:22.620245 11118 log.cc:824] T bf22324b81b847d2b593830e1d070bc5 P 35a9659542c04f3a85eef35cd6488d4c: Log is configured to *not* fsync() on all Append() calls
I20260924 04:09:22.622190 11118 tablet_bootstrap.cc:492] T bf22324b81b847d2b593830e1d070bc5 P 35a9659542c04f3a85eef35cd6488d4c: No bootstrap required, opened a new log
I20260924 04:09:22.622359 11118 ts_tablet_manager.cc:1397] T bf22324b81b847d2b593830e1d070bc5 P 35a9659542c04f3a85eef35cd6488d4c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 04:09:22.622381 11117 raft_consensus.cc:348] T bf22324b81b847d2b593830e1d070bc5 P c0c47ee86ebe4ea78d331fd0982bb610 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0c47ee86ebe4ea78d331fd0982bb610" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34013 } } peers { permanent_uuid: "95efe778539a4c61b9e39e1d058ecc48" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35495 } } peers { permanent_uuid: "35a9659542c04f3a85eef35cd6488d4c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45595 } }
I20260924 04:09:22.622668 11117 raft_consensus.cc:374] T bf22324b81b847d2b593830e1d070bc5 P c0c47ee86ebe4ea78d331fd0982bb610 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:09:22.622727 11117 raft_consensus.cc:729] T bf22324b81b847d2b593830e1d070bc5 P c0c47ee86ebe4ea78d331fd0982bb610 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0c47ee86ebe4ea78d331fd0982bb610, State: Initialized, Role: FOLLOWER
I20260924 04:09:22.623216 11117 consensus_queue.cc:260] T bf22324b81b847d2b593830e1d070bc5 P c0c47ee86ebe4ea78d331fd0982bb610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0c47ee86ebe4ea78d331fd0982bb610" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34013 } } peers { permanent_uuid: "95efe778539a4c61b9e39e1d058ecc48" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35495 } } peers { permanent_uuid: "35a9659542c04f3a85eef35cd6488d4c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45595 } }
I20260924 04:09:22.623193 11116 raft_consensus.cc:348] T bf22324b81b847d2b593830e1d070bc5 P 95efe778539a4c61b9e39e1d058ecc48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0c47ee86ebe4ea78d331fd0982bb610" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34013 } } peers { permanent_uuid: "95efe778539a4c61b9e39e1d058ecc48" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35495 } } peers { permanent_uuid: "35a9659542c04f3a85eef35cd6488d4c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45595 } }
I20260924 04:09:22.623425 11116 raft_consensus.cc:374] T bf22324b81b847d2b593830e1d070bc5 P 95efe778539a4c61b9e39e1d058ecc48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:09:22.623471 11116 raft_consensus.cc:729] T bf22324b81b847d2b593830e1d070bc5 P 95efe778539a4c61b9e39e1d058ecc48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95efe778539a4c61b9e39e1d058ecc48, State: Initialized, Role: FOLLOWER
I20260924 04:09:22.623850 11116 consensus_queue.cc:260] T bf22324b81b847d2b593830e1d070bc5 P 95efe778539a4c61b9e39e1d058ecc48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0c47ee86ebe4ea78d331fd0982bb610" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34013 } } peers { permanent_uuid: "95efe778539a4c61b9e39e1d058ecc48" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35495 } } peers { permanent_uuid: "35a9659542c04f3a85eef35cd6488d4c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45595 } }
I20260924 04:09:22.624184 11117 ts_tablet_manager.cc:1428] T bf22324b81b847d2b593830e1d070bc5 P c0c47ee86ebe4ea78d331fd0982bb610: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
I20260924 04:09:22.624511 10965 heartbeater.cc:503] Master 127.0.98.62:35965 was elected leader, sending a full tablet report...
I20260924 04:09:22.624751 11116 ts_tablet_manager.cc:1428] T bf22324b81b847d2b593830e1d070bc5 P 95efe778539a4c61b9e39e1d058ecc48: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260924 04:09:22.625487 11099 heartbeater.cc:503] Master 127.0.98.62:35965 was elected leader, sending a full tablet report...
I20260924 04:09:22.625840 11118 raft_consensus.cc:348] T bf22324b81b847d2b593830e1d070bc5 P 35a9659542c04f3a85eef35cd6488d4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0c47ee86ebe4ea78d331fd0982bb610" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34013 } } peers { permanent_uuid: "95efe778539a4c61b9e39e1d058ecc48" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35495 } } peers { permanent_uuid: "35a9659542c04f3a85eef35cd6488d4c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45595 } }
I20260924 04:09:22.626060 11118 raft_consensus.cc:374] T bf22324b81b847d2b593830e1d070bc5 P 35a9659542c04f3a85eef35cd6488d4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:09:22.626113 11118 raft_consensus.cc:729] T bf22324b81b847d2b593830e1d070bc5 P 35a9659542c04f3a85eef35cd6488d4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35a9659542c04f3a85eef35cd6488d4c, State: Initialized, Role: FOLLOWER
I20260924 04:09:22.626535 11118 consensus_queue.cc:260] T bf22324b81b847d2b593830e1d070bc5 P 35a9659542c04f3a85eef35cd6488d4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0c47ee86ebe4ea78d331fd0982bb610" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34013 } } peers { permanent_uuid: "95efe778539a4c61b9e39e1d058ecc48" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35495 } } peers { permanent_uuid: "35a9659542c04f3a85eef35cd6488d4c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45595 } }
I20260924 04:09:22.627215 10831 heartbeater.cc:503] Master 127.0.98.62:35965 was elected leader, sending a full tablet report...
I20260924 04:09:22.627439 11118 ts_tablet_manager.cc:1428] T bf22324b81b847d2b593830e1d070bc5 P 35a9659542c04f3a85eef35cd6488d4c: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260924 04:09:22.663779 11124 raft_consensus.cc:482] T bf22324b81b847d2b593830e1d070bc5 P 35a9659542c04f3a85eef35cd6488d4c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:09:22.663998 11124 raft_consensus.cc:504] T bf22324b81b847d2b593830e1d070bc5 P 35a9659542c04f3a85eef35cd6488d4c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0c47ee86ebe4ea78d331fd0982bb610" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34013 } } peers { permanent_uuid: "95efe778539a4c61b9e39e1d058ecc48" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35495 } } peers { permanent_uuid: "35a9659542c04f3a85eef35cd6488d4c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45595 } }
I20260924 04:09:22.665318 11124 leader_election.cc:288] T bf22324b81b847d2b593830e1d070bc5 P 35a9659542c04f3a85eef35cd6488d4c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0c47ee86ebe4ea78d331fd0982bb610 (127.0.98.3:34013), 95efe778539a4c61b9e39e1d058ecc48 (127.0.98.2:35495)
I20260924 04:09:22.669374 11114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:09:22.665736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59763 (local address 127.0.98.2:35495)
0924 04:09:22.665874 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 04:09:22.665880 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:09:22.666230 (+ 350us) server_negotiation.cc:407] Connection header received
0924 04:09:22.666352 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:09:22.666355 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:09:22.666404 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:09:22.666482 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:09:22.666755 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:22.667266 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:22.668321 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:22.668800 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:22.668880 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:22.668953 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 04:09:22.669159 (+ 206us) server_negotiation.cc:299] Negotiation successful
0924 04:09:22.669240 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 04:09:22.669409 11111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:09:22.665793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36641 (local address 127.0.98.3:34013)
0924 04:09:22.665950 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 04:09:22.665954 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:09:22.666086 (+ 132us) server_negotiation.cc:407] Connection header received
0924 04:09:22.666212 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:09:22.666215 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:09:22.666269 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:09:22.666351 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:09:22.666634 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:22.667171 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:22.668387 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:22.668876 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:22.668978 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:22.669035 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 04:09:22.669237 (+ 202us) server_negotiation.cc:299] Negotiation successful
0924 04:09:22.669306 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260924 04:09:22.669420 11125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:09:22.665630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35495 (local address 127.0.98.1:59763)
0924 04:09:22.666118 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0924 04:09:22.666130 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:09:22.666237 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:09:22.666506 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:09:22.666508 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:09:22.666521 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:09:22.666640 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:22.666646 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:22.667371 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:22.667373 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:22.668208 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:22.668216 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:22.668912 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:22.668919 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:22.669026 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:22.669052 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 04:09:22.669132 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 04:09:22.669229 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406,"mutex_wait_us":4,"thread_start_us":101,"threads_started":1}
I20260924 04:09:22.669428 11126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:09:22.665616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34013 (local address 127.0.98.1:36641)
0924 04:09:22.665995 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0924 04:09:22.666006 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:09:22.666102 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:09:22.666365 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:09:22.666368 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:09:22.666382 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:09:22.666525 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:22.666532 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:22.667274 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:22.667277 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:22.668257 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:09:22.668286 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:09:22.668998 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:09:22.669001 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:09:22.669087 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:22.669133 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 04:09:22.669210 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 04:09:22.669271 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"mutex_wait_us":21,"thread_start_us":123,"threads_started":1}
I20260924 04:09:22.670058 10919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf22324b81b847d2b593830e1d070bc5" candidate_uuid: "35a9659542c04f3a85eef35cd6488d4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95efe778539a4c61b9e39e1d058ecc48" is_pre_election: true
I20260924 04:09:22.670105 11053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf22324b81b847d2b593830e1d070bc5" candidate_uuid: "35a9659542c04f3a85eef35cd6488d4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0c47ee86ebe4ea78d331fd0982bb610" is_pre_election: true
I20260924 04:09:22.670403 11053 raft_consensus.cc:2454] T bf22324b81b847d2b593830e1d070bc5 P c0c47ee86ebe4ea78d331fd0982bb610 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35a9659542c04f3a85eef35cd6488d4c in term 0.
I20260924 04:09:22.670408 10919 raft_consensus.cc:2454] T bf22324b81b847d2b593830e1d070bc5 P 95efe778539a4c61b9e39e1d058ecc48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35a9659542c04f3a85eef35cd6488d4c in term 0.
I20260924 04:09:22.670876 10720 leader_election.cc:302] T bf22324b81b847d2b593830e1d070bc5 P 35a9659542c04f3a85eef35cd6488d4c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 35a9659542c04f3a85eef35cd6488d4c, 95efe778539a4c61b9e39e1d058ecc48, c0c47ee86ebe4ea78d331fd0982bb610; no voters:
I20260924 04:09:22.671154 11124 raft_consensus.cc:2781] T bf22324b81b847d2b593830e1d070bc5 P 35a9659542c04f3a85eef35cd6488d4c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:09:22.671229 11124 raft_consensus.cc:482] T bf22324b81b847d2b593830e1d070bc5 P 35a9659542c04f3a85eef35cd6488d4c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:09:22.671259 11124 raft_consensus.cc:3037] T bf22324b81b847d2b593830e1d070bc5 P 35a9659542c04f3a85eef35cd6488d4c [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:09:22.672250 11124 raft_consensus.cc:504] T bf22324b81b847d2b593830e1d070bc5 P 35a9659542c04f3a85eef35cd6488d4c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0c47ee86ebe4ea78d331fd0982bb610" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34013 } } peers { permanent_uuid: "95efe778539a4c61b9e39e1d058ecc48" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35495 } } peers { permanent_uuid: "35a9659542c04f3a85eef35cd6488d4c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45595 } }
I20260924 04:09:22.672585 11124 leader_election.cc:288] T bf22324b81b847d2b593830e1d070bc5 P 35a9659542c04f3a85eef35cd6488d4c [CANDIDATE]: Term 1 election: Requested vote from peers c0c47ee86ebe4ea78d331fd0982bb610 (127.0.98.3:34013), 95efe778539a4c61b9e39e1d058ecc48 (127.0.98.2:35495)
I20260924 04:09:22.672955 11053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf22324b81b847d2b593830e1d070bc5" candidate_uuid: "35a9659542c04f3a85eef35cd6488d4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0c47ee86ebe4ea78d331fd0982bb610"
I20260924 04:09:22.673026 10919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf22324b81b847d2b593830e1d070bc5" candidate_uuid: "35a9659542c04f3a85eef35cd6488d4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95efe778539a4c61b9e39e1d058ecc48"
I20260924 04:09:22.673066 11053 raft_consensus.cc:3037] T bf22324b81b847d2b593830e1d070bc5 P c0c47ee86ebe4ea78d331fd0982bb610 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:09:22.673177 10919 raft_consensus.cc:3037] T bf22324b81b847d2b593830e1d070bc5 P 95efe778539a4c61b9e39e1d058ecc48 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:09:22.674247 10919 raft_consensus.cc:2454] T bf22324b81b847d2b593830e1d070bc5 P 95efe778539a4c61b9e39e1d058ecc48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35a9659542c04f3a85eef35cd6488d4c in term 1.
I20260924 04:09:22.674260 11053 raft_consensus.cc:2454] T bf22324b81b847d2b593830e1d070bc5 P c0c47ee86ebe4ea78d331fd0982bb610 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35a9659542c04f3a85eef35cd6488d4c in term 1.
I20260924 04:09:22.674556 10718 leader_election.cc:302] T bf22324b81b847d2b593830e1d070bc5 P 35a9659542c04f3a85eef35cd6488d4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35a9659542c04f3a85eef35cd6488d4c, c0c47ee86ebe4ea78d331fd0982bb610; no voters:
I20260924 04:09:22.674747 11124 raft_consensus.cc:2781] T bf22324b81b847d2b593830e1d070bc5 P 35a9659542c04f3a85eef35cd6488d4c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:09:22.674935 11124 raft_consensus.cc:686] T bf22324b81b847d2b593830e1d070bc5 P 35a9659542c04f3a85eef35cd6488d4c [term 1 LEADER]: Becoming Leader. State: Replica: 35a9659542c04f3a85eef35cd6488d4c, State: Running, Role: LEADER
I20260924 04:09:22.675274 11124 consensus_queue.cc:237] T bf22324b81b847d2b593830e1d070bc5 P 35a9659542c04f3a85eef35cd6488d4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0c47ee86ebe4ea78d331fd0982bb610" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34013 } } peers { permanent_uuid: "95efe778539a4c61b9e39e1d058ecc48" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35495 } } peers { permanent_uuid: "35a9659542c04f3a85eef35cd6488d4c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45595 } }
I20260924 04:09:22.678452 10638 catalog_manager.cc:5495] T bf22324b81b847d2b593830e1d070bc5 P 35a9659542c04f3a85eef35cd6488d4c reported cstate change: term changed from 0 to 1, leader changed from <none> to 35a9659542c04f3a85eef35cd6488d4c (127.0.98.1). New cstate: current_term: 1 leader_uuid: "35a9659542c04f3a85eef35cd6488d4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0c47ee86ebe4ea78d331fd0982bb610" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34013 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95efe778539a4c61b9e39e1d058ecc48" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35a9659542c04f3a85eef35cd6488d4c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45595 } health_report { overall_health: HEALTHY } } }
I20260924 04:09:22.698125 11115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:09:22.694637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58718 (local address 127.0.98.1:45595)
0924 04:09:22.694785 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 04:09:22.694789 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:09:22.694803 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:09:22.694875 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:09:22.694878 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:09:22.694934 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:09:22.695049 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:09:22.695273 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:22.695823 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:22.696750 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:09:22.696975 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:09:22.697058 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:09:22.697354 (+ 296us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:09:22.697792 (+ 438us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:09:22.697897 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 04:09:22.697945 (+ 48us) server_negotiation.cc:299] Negotiation successful
0924 04:09:22.698013 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 04:09:22.704181 10919 raft_consensus.cc:1261] T bf22324b81b847d2b593830e1d070bc5 P 95efe778539a4c61b9e39e1d058ecc48 [term 1 FOLLOWER]: Refusing update from remote peer 35a9659542c04f3a85eef35cd6488d4c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:09:22.704190 11053 raft_consensus.cc:1261] T bf22324b81b847d2b593830e1d070bc5 P c0c47ee86ebe4ea78d331fd0982bb610 [term 1 FOLLOWER]: Refusing update from remote peer 35a9659542c04f3a85eef35cd6488d4c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:09:22.704914 11127 consensus_queue.cc:1035] T bf22324b81b847d2b593830e1d070bc5 P 35a9659542c04f3a85eef35cd6488d4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0c47ee86ebe4ea78d331fd0982bb610" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:09:22.705123 11124 consensus_queue.cc:1035] T bf22324b81b847d2b593830e1d070bc5 P 35a9659542c04f3a85eef35cd6488d4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "95efe778539a4c61b9e39e1d058ecc48" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:09:22.711972 11130 mvcc.cc:205] Tried to move back new op lower bound from 7332753255228858368 to 7332753255120506880. Current Snapshot: MvccSnapshot[applied={T|T < 7332753255228858368}]
I20260924 04:09:22.712584 11131 mvcc.cc:205] Tried to move back new op lower bound from 7332753255228858368 to 7332753255120506880. Current Snapshot: MvccSnapshot[applied={T|T < 7332753255228858368}]
I20260924 04:09:22.712776 11132 mvcc.cc:205] Tried to move back new op lower bound from 7332753255228858368 to 7332753255120506880. Current Snapshot: MvccSnapshot[applied={T|T < 7332753255228858368}]
I20260924 04:09:22.722095 10639 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44648:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:09:22.722249 10639 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44648:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:09:22.724660 10639 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 24a6017fc4344885a8a392ceb9496eac: Sending DeleteTablet for 3 replicas of tablet bf22324b81b847d2b593830e1d070bc5
I20260924 04:09:22.725510 11033 tablet_service.cc:1514] Processing DeleteTablet for tablet bf22324b81b847d2b593830e1d070bc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:09:22 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58734
I20260924 04:09:22.725503 10899 tablet_service.cc:1514] Processing DeleteTablet for tablet bf22324b81b847d2b593830e1d070bc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:09:22 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59686
I20260924 04:09:22.725524 10765 tablet_service.cc:1514] Processing DeleteTablet for tablet bf22324b81b847d2b593830e1d070bc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:09:22 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58716
I20260924 04:09:22.725975 11142 tablet_replica.cc:330] T bf22324b81b847d2b593830e1d070bc5 P 35a9659542c04f3a85eef35cd6488d4c: stopping tablet replica
I20260924 04:09:22.726179 11140 tablet_replica.cc:330] T bf22324b81b847d2b593830e1d070bc5 P c0c47ee86ebe4ea78d331fd0982bb610: stopping tablet replica
I20260924 04:09:22.726430 11142 raft_consensus.cc:2229] T bf22324b81b847d2b593830e1d070bc5 P 35a9659542c04f3a85eef35cd6488d4c [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:09:22.726604 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 10702
I20260924 04:09:22.726727 11140 raft_consensus.cc:2229] T bf22324b81b847d2b593830e1d070bc5 P c0c47ee86ebe4ea78d331fd0982bb610 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:09:22.726822 11142 raft_consensus.cc:2258] T bf22324b81b847d2b593830e1d070bc5 P 35a9659542c04f3a85eef35cd6488d4c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:09:22.727284 11140 raft_consensus.cc:2258] T bf22324b81b847d2b593830e1d070bc5 P c0c47ee86ebe4ea78d331fd0982bb610 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:09:22.729555 11140 ts_tablet_manager.cc:1905] T bf22324b81b847d2b593830e1d070bc5 P c0c47ee86ebe4ea78d331fd0982bb610: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:09:22.731689 11140 ts_tablet_manager.cc:1918] T bf22324b81b847d2b593830e1d070bc5 P c0c47ee86ebe4ea78d331fd0982bb610: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:09:22.731768 11140 log.cc:1196] T bf22324b81b847d2b593830e1d070bc5 P c0c47ee86ebe4ea78d331fd0982bb610: Deleting WAL directory at /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790222898807720-392-0/minicluster-data/ts-2/wal/wals/bf22324b81b847d2b593830e1d070bc5
I20260924 04:09:22.732025 11141 tablet_replica.cc:330] T bf22324b81b847d2b593830e1d070bc5 P 95efe778539a4c61b9e39e1d058ecc48: stopping tablet replica
I20260924 04:09:22.732116 11140 ts_tablet_manager.cc:1939] T bf22324b81b847d2b593830e1d070bc5 P c0c47ee86ebe4ea78d331fd0982bb610: Deleting consensus metadata
I20260924 04:09:22.732372 11141 raft_consensus.cc:2229] T bf22324b81b847d2b593830e1d070bc5 P 95efe778539a4c61b9e39e1d058ecc48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:09:22.732699 11141 raft_consensus.cc:2258] T bf22324b81b847d2b593830e1d070bc5 P 95efe778539a4c61b9e39e1d058ecc48 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260924 04:09:22.733132 10624 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45595 (error 108)
I20260924 04:09:22.733625 10624 catalog_manager.cc:4841] TS c0c47ee86ebe4ea78d331fd0982bb610 (127.0.98.3:34013): tablet bf22324b81b847d2b593830e1d070bc5 (table test-table [id=92a98efb912a4026bf565c534b1af187]) successfully deleted
I20260924 04:09:22.734191 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 10836
I20260924 04:09:22.734221 11112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:09:22.733927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45595 (local address 127.0.0.1:58734)
0924 04:09:22.734078 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 04:09:22.734153 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260924 04:09:22.734261 11141 ts_tablet_manager.cc:1905] T bf22324b81b847d2b593830e1d070bc5 P 95efe778539a4c61b9e39e1d058ecc48: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 04:09:22.735668 10624 catalog_manager.cc:4584] TS 35a9659542c04f3a85eef35cd6488d4c (127.0.98.1:45595): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bf22324b81b847d2b593830e1d070bc5: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45595: connect: Connection refused (error 111)
I20260924 04:09:22.740180 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 10970
I20260924 04:09:22.741238 11112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:09:22.740457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35495 (local address 127.0.0.1:59700)
0924 04:09:22.741058 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0924 04:09:22.741141 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:35495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":503}
I20260924 04:09:22.745783 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 10608
2026-09-24T04:09:22Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2247 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:09:22 dist-test-slave-00ld krb5kdc[11146](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:09:22 dist-test-slave-00ld krb5kdc[11146](info): set up 2 sockets
Sep 24 04:09:22 dist-test-slave-00ld krb5kdc[11146](info): commencing operation
krb5kdc: starting...
W20260924 04:09:23.793728 392 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/krb5kdc/test-user.keytab.
Sep 24 04:09:23 dist-test-slave-00ld krb5kdc[11146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222963, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:09:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:09:23Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260924 04:09:23.959630 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task1CVu5m/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/minicluster-data/postgres -l logfile start
2026-09-24 04:09:26.489 UTC [11176] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 04:09:26.489 UTC [11176] LOG: listening on IPv4 address "127.0.98.20", port 51397
2026-09-24 04:09:26.493 UTC [11176] LOG: listening on Unix socket "/tmp/.s.PGSQL.51397"
2026-09-24 04:09:26.514 UTC [11179] LOG: database system was shut down at 2026-09-24 04:09:25 UTC
2026-09-24 04:09:26.518 UTC [11176] LOG: database system is ready to accept connections
I20260924 04:09:27.469669 392 mini_postgres.cc:96] Postgres bound to 51397
2026-09-24 04:09:27.476 UTC [11187] FATAL: database "slave" does not exist
127.0.98.20:51397 - accepting connections
I20260924 04:09:27.476637 392 mini_ranger.cc:161] Starting Ranger...
I20260924 04:09:27.491410 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 04:09:28.001395 392 mini_ranger.cc:87] Created ranger Postgres database
I20260924 04:09:28.001505 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/minicluster-data/ranger-admin
2026-09-24 04:09:28,295 [I] DB FLAVOR :POSTGRES
2026-09-24 04:09:28,296 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 04:09:28,296 [I] Checking connection..
2026-09-24 04:09:28,296 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1CVu5m/build/debug/bin/postgresql.jar:/tmp/dist-test-task1CVu5m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51397/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 04:09:28,542 [I] Checking connection passed.
2026-09-24 04:09:28,542 [I] --------- Verifying version history table ---------
2026-09-24 04:09:28,543 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1CVu5m/build/debug/bin/postgresql.jar:/tmp/dist-test-task1CVu5m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51397/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 04:09:28,803 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 04:09:28,803 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1CVu5m/build/debug/bin/postgresql.jar:/tmp/dist-test-task1CVu5m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51397/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 04:09:29,062 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 04:09:29,062 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 04:09:29,063 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1CVu5m/build/debug/bin/postgresql.jar:/tmp/dist-test-task1CVu5m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51397/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task1CVu5m/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 04:09:29.303 UTC [11251] WARNING: there is no transaction in progress
2026-09-24 04:09:29,309 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 04:09:29,309 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1CVu5m/build/debug/bin/postgresql.jar:/tmp/dist-test-task1CVu5m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51397/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 04:09:29,568 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 04:09:29,568 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 04:09:29,568 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1CVu5m/build/debug/bin/postgresql.jar:/tmp/dist-test-task1CVu5m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51397/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 04:09:29,825 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1CVu5m/build/debug/bin/postgresql.jar:/tmp/dist-test-task1CVu5m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51397/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 04:09:30,099 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1CVu5m/build/debug/bin/postgresql.jar:/tmp/dist-test-task1CVu5m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51397/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-00ld.c.gcp-upstream.internal','N') ;"
2026-09-24 04:09:30,320 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1CVu5m/build/debug/bin/postgresql.jar:/tmp/dist-test-task1CVu5m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51397/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 04:09:30,552 [I] Table x_portal_user does not exist in database ranger
2026-09-24 04:09:30,552 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1CVu5m/build/debug/bin/postgresql.jar:/tmp/dist-test-task1CVu5m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51397/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 04:09:30,809 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 04:09:30,809 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1CVu5m/build/debug/bin/postgresql.jar:/tmp/dist-test-task1CVu5m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51397/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 04:09:31,058 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1CVu5m/build/debug/bin/postgresql.jar:/tmp/dist-test-task1CVu5m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51397/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 04:09:31,309 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 04:09:31,309 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1CVu5m/build/debug/bin/postgresql.jar:/tmp/dist-test-task1CVu5m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51397/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task1CVu5m/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 04:09:32.281 UTC [11359] WARNING: there is no transaction in progress
2026-09-24 04:09:32.297 UTC [11359] WARNING: there is no transaction in progress
2026-09-24 04:09:32.313 UTC [11359] WARNING: there is no transaction in progress
2026-09-24 04:09:32.329 UTC [11359] WARNING: there is no transaction in progress
2026-09-24 04:09:32.347 UTC [11359] WARNING: there is no transaction in progress
2026-09-24 04:09:32.364 UTC [11359] WARNING: there is no transaction in progress
2026-09-24 04:09:32.418 UTC [11359] WARNING: there is no transaction in progress
2026-09-24 04:09:32.425 UTC [11359] WARNING: there is no transaction in progress
2026-09-24 04:09:32.439 UTC [11359] WARNING: there is no transaction in progress
2026-09-24 04:09:32.449 UTC [11359] WARNING: there is no transaction in progress
2026-09-24 04:09:32.459 UTC [11359] WARNING: there is no transaction in progress
2026-09-24 04:09:32.474 UTC [11359] WARNING: there is no transaction in progress
2026-09-24 04:09:32.483 UTC [11359] WARNING: there is no transaction in progress
2026-09-24 04:09:32.490 UTC [11359] WARNING: there is no transaction in progress
2026-09-24 04:09:32.497 UTC [11359] WARNING: there is no transaction in progress
2026-09-24 04:09:32.503 UTC [11359] WARNING: there is no transaction in progress
2026-09-24 04:09:33,159 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 04:09:33,159 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1CVu5m/build/debug/bin/postgresql.jar:/tmp/dist-test-task1CVu5m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51397/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 04:09:33,395 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 04:09:33,395 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1CVu5m/build/debug/bin/postgresql.jar:/tmp/dist-test-task1CVu5m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51397/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 04:09:33,658 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 04:09:33,658 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1CVu5m/build/debug/bin/postgresql.jar:/tmp/dist-test-task1CVu5m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51397/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 04:09:33,917 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 04:09:33,917 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1CVu5m/build/debug/bin/postgresql.jar:/tmp/dist-test-task1CVu5m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51397/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 04:09:34,151 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1CVu5m/build/debug/bin/postgresql.jar:/tmp/dist-test-task1CVu5m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51397/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 04:09:34,402 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1CVu5m/build/debug/bin/postgresql.jar:/tmp/dist-test-task1CVu5m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51397/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-00ld.c.gcp-upstream.internal';"
2026-09-24 04:09:34,647 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 04:09:34,647 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1CVu5m/build/debug/bin/postgresql.jar:/tmp/dist-test-task1CVu5m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51397/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 04:09:34,886 [I] DB_PATCHES have already been applied
I20260924 04:09:34.891433 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/minicluster-data/ranger-admin:/tmp/dist-test-task1CVu5m/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task1CVu5m/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task1CVu5m/build/debug/bin/postgresql.jar:/tmp/dist-test-task1CVu5m/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 04:09:34.891517 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260924 04:09:34.894841 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:55397
Sep 24, 2026 4:09:35 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 4:09:35 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 4:09:35 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 4:09:35 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 4:09:35 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 4:09:35 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 4:09:35 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 04:09:35.999014 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.522s user 0.007s sys 0.002s
Sep 24, 2026 4:09:55 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 4:09:55 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 24, 2026 4:09:55 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 4:09:55 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 4:09:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 4:09:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 4:09:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 4:09:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 4:09:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 4:09:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 4:09:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 4:09:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 4:09:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 4:09:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 4:09:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 4:09:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 4:09:55 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 4:09:56 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 04:09:57.459870 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 04:09:57 dist-test-slave-00ld krb5kdc[11146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222997, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 04:09:57.544780 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 04:09:57.553778 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 04:09:58.442139 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 04:09:58.442224 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/minicluster-data/ranger-kms
I20260924 04:09:58.442320 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:51397
2026-09-24 04:09:58,632 [I] DB FLAVOR :POSTGRES
2026-09-24 04:09:58,632 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 04:09:58,633 [I] Checking connection
2026-09-24 04:09:58,633 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1CVu5m/build/debug/bin/postgresql.jar:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51397/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 04:09:58,885 [I] connection success
2026-09-24 04:09:58,885 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 04:09:58,885 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 04:09:58,886 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1CVu5m/build/debug/bin/postgresql.jar:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51397/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 04:09:59,121 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 04:09:59,121 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 04:09:59,121 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 04:09:59,121 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1CVu5m/build/debug/bin/postgresql.jar:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51397/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 04:09:59,406 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 04:09:59.859562 392 mini_ranger_kms.cc:310] Created kms service
I20260924 04:10:00.030318 392 mini_ranger_kms.cc:326] Created kudu user
I20260924 04:10:00.076416 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 04:10:00 dist-test-slave-00ld krb5kdc[11146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790222997, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 04:10:00.477202 392 mini_ranger_kms.cc:384] Added ranger policy
I20260924 04:10:00.477352 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/minicluster-data/ranger-kms:/tmp/dist-test-task1CVu5m/build/debug/bin/postgresql.jar:/tmp/dist-test-task1CVu5m/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task1CVu5m/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 04:10:00.477378 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260924 04:10:00.481402 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11539
I20260924 04:10:00.481531 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:54703
Sep 24, 2026 4:10:01 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 4:10:01 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 4:10:01 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 4:10:01 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 4:10:02 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 4:10:02 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260924 04:10:02.780937 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.236s user 0.007s sys 0.166s
I20260924 04:10:02.781088 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 04:10:02.781229 392 mini_ranger_kms.cc:401] 127.0.98.20:54703/kms/v1/keys
Sep 24 04:10:02 dist-test-slave-00ld krb5kdc[11146](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790223002, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 24 04:10:03 dist-test-slave-00ld krb5kdc[11146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223002, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 24 04:10:03 dist-test-slave-00ld krb5kdc[11146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223002, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 24 04:10:06 dist-test-slave-00ld krb5kdc[11146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223006, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:10:06.115621 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37555
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37555
--ranger_config_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:54703/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:10:06.226776 11590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:10:06.227015 11590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:10:06.227052 11590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:10:06.231070 11590 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:10:06.231130 11590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:10:06.231151 11590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:10:06.231171 11590 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:10:06.231187 11590 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:10:06.236584 11590 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:54703/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37555
--ranger_config_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37555
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11590
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:10:06.237645 11590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:10:06.238675 11590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:10:06.247324 11595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:10:06.247337 11598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:10:06.247325 11596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:10:06.247541 11590 server_base.cc:1034] running on GCE node
I20260924 04:10:06.248186 11590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:10:06.249358 11590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:10:06.250551 11590 hybrid_clock.cc:648] HybridClock initialized: now 1790223006250530 us; error 42 us; skew 500 ppm
Sep 24 04:10:06 dist-test-slave-00ld krb5kdc[11146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223006, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:10:06.259785 11590 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:10:06.260964 11590 webserver.cc:460] Webserver started at http://127.0.98.62:33841/ using document root <none> and password file <none>
I20260924 04:10:06.261541 11590 fs_manager.cc:359] Metadata directory not provided
I20260924 04:10:06.261631 11590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:10:06.261842 11590 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 04:10:06 dist-test-slave-00ld krb5kdc[11146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223006, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 04:10:06.307379 11590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/minicluster-data/master-0/data/instance:
uuid: "3f488253ca6f40a9bb4747bbc564a018"
format_stamp: "Formatted at 2026-09-24 04:10:06 on dist-test-slave-00ld"
server_key: "227220d00424e7ea6f665e228ee53609"
server_key_iv: "bf6920667299f7a6e2404dddcae55530"
server_key_version: "kuduclusterkey@0"
I20260924 04:10:06.308008 11590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/minicluster-data/master-0/wal/instance:
uuid: "3f488253ca6f40a9bb4747bbc564a018"
format_stamp: "Formatted at 2026-09-24 04:10:06 on dist-test-slave-00ld"
server_key: "227220d00424e7ea6f665e228ee53609"
server_key_iv: "bf6920667299f7a6e2404dddcae55530"
server_key_version: "kuduclusterkey@0"
I20260924 04:10:06.311915 11590 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 04:10:06.325821 11605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:10:06.327296 11590 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 04:10:06.327404 11590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/minicluster-data/master-0/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/minicluster-data/master-0/wal
uuid: "3f488253ca6f40a9bb4747bbc564a018"
format_stamp: "Formatted at 2026-09-24 04:10:06 on dist-test-slave-00ld"
server_key: "227220d00424e7ea6f665e228ee53609"
server_key_iv: "bf6920667299f7a6e2404dddcae55530"
server_key_version: "kuduclusterkey@0"
I20260924 04:10:06.327556 11590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:10:06.345422 11590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:10:06.348349 11590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:10:06.348557 11590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:10:06.356647 11590 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37555
I20260924 04:10:06.356705 11657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37555 every 8 connection(s)
I20260924 04:10:06.357685 11590 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:10:06.360405 11658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:10:06.362928 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 11590
I20260924 04:10:06.363052 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/minicluster-data/master-0/wal/instance
I20260924 04:10:06.363353 392 external_mini_cluster.cc:1428] Setting key 08580afa2e0ecdc0454c7408a4cf1c23
I20260924 04:10:06.366520 11658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f488253ca6f40a9bb4747bbc564a018: Bootstrap starting.
I20260924 04:10:06.369134 11658 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f488253ca6f40a9bb4747bbc564a018: Neither blocks nor log segments found. Creating new log.
I20260924 04:10:06.369886 11658 log.cc:824] T 00000000000000000000000000000000 P 3f488253ca6f40a9bb4747bbc564a018: Log is configured to *not* fsync() on all Append() calls
I20260924 04:10:06.372174 11658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f488253ca6f40a9bb4747bbc564a018: No bootstrap required, opened a new log
Sep 24 04:10:06 dist-test-slave-00ld krb5kdc[11146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223006, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:10:06.376340 11658 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f488253ca6f40a9bb4747bbc564a018 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f488253ca6f40a9bb4747bbc564a018" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37555 } }
I20260924 04:10:06.376637 11658 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3f488253ca6f40a9bb4747bbc564a018 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:10:06.376868 11658 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f488253ca6f40a9bb4747bbc564a018 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f488253ca6f40a9bb4747bbc564a018, State: Initialized, Role: FOLLOWER
I20260924 04:10:06.377452 11658 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f488253ca6f40a9bb4747bbc564a018 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f488253ca6f40a9bb4747bbc564a018" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37555 } }
I20260924 04:10:06.377648 11658 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f488253ca6f40a9bb4747bbc564a018 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:10:06.377712 11658 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f488253ca6f40a9bb4747bbc564a018 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:10:06.377810 11658 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f488253ca6f40a9bb4747bbc564a018 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:10:06.378288 11661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:10:06.364834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43642 (local address 127.0.98.62:37555)
0924 04:10:06.365290 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0924 04:10:06.365312 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0924 04:10:06.365356 (+ 44us) server_negotiation.cc:407] Connection header received
0924 04:10:06.366130 (+ 774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:10:06.366151 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:10:06.366548 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:10:06.366906 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:10:06.367745 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:10:06.369206 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:10:06.370442 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:10:06.370960 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:10:06.371139 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:10:06.373663 (+ 2524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:10:06.373669 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:10:06.373690 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:10:06.373715 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:10:06.375265 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:10:06.375691 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:10:06.375701 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:10:06.375705 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:10:06.375799 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:10:06.376090 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:10:06.376093 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:10:06.376095 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:10:06.376453 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:10:06.376750 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0924 04:10:06.377064 (+ 314us) server_negotiation.cc:299] Negotiation successful
0924 04:10:06.377316 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":133,"threads_started":1}
I20260924 04:10:06.378851 11658 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f488253ca6f40a9bb4747bbc564a018 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f488253ca6f40a9bb4747bbc564a018" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37555 } }
I20260924 04:10:06.379205 11658 leader_election.cc:302] T 00000000000000000000000000000000 P 3f488253ca6f40a9bb4747bbc564a018 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f488253ca6f40a9bb4747bbc564a018; no voters:
I20260924 04:10:06.379556 11658 leader_election.cc:288] T 00000000000000000000000000000000 P 3f488253ca6f40a9bb4747bbc564a018 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:10:06.379648 11663 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f488253ca6f40a9bb4747bbc564a018 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:10:06.379894 11663 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f488253ca6f40a9bb4747bbc564a018 [term 1 LEADER]: Becoming Leader. State: Replica: 3f488253ca6f40a9bb4747bbc564a018, State: Running, Role: LEADER
I20260924 04:10:06.380223 11663 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f488253ca6f40a9bb4747bbc564a018 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f488253ca6f40a9bb4747bbc564a018" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37555 } }
I20260924 04:10:06.380873 11658 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f488253ca6f40a9bb4747bbc564a018 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:10:06.381597 11663 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f488253ca6f40a9bb4747bbc564a018 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3f488253ca6f40a9bb4747bbc564a018" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f488253ca6f40a9bb4747bbc564a018" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37555 } } }
I20260924 04:10:06.381778 11663 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f488253ca6f40a9bb4747bbc564a018 [sys.catalog]: This master's current role is: LEADER
I20260924 04:10:06.382010 11663 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f488253ca6f40a9bb4747bbc564a018 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f488253ca6f40a9bb4747bbc564a018. Latest consensus state: current_term: 1 leader_uuid: "3f488253ca6f40a9bb4747bbc564a018" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f488253ca6f40a9bb4747bbc564a018" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37555 } } }
I20260924 04:10:06.382143 11663 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f488253ca6f40a9bb4747bbc564a018 [sys.catalog]: This master's current role is: LEADER
I20260924 04:10:06.382881 11658 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 04:10:06.386155 11672 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:10:06.390239 11672 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:10:06.396275 11672 catalog_manager.cc:1347] Generated new cluster ID: cf94c4b1c4524a0db332654ff319ca1e
I20260924 04:10:06.396341 11672 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:10:06.410547 11672 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:10:06.411834 11672 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:10:06.416669 11672 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3f488253ca6f40a9bb4747bbc564a018: Generated new TSK 0
I20260924 04:10:06.417464 11672 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:10:07.207937 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:54703/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37555
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:10:07.368376 11703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:10:07.368817 11703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:10:07.368932 11703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:10:07.375138 11703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:10:07.375227 11703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:10:07.375327 11703 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:10:07.380887 11703 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:54703/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37555
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11703
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:10:07.382197 11703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:10:07.383522 11703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:10:07.395726 11710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:10:07.395828 11713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:10:07.395735 11711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:10:07.396162 11703 server_base.cc:1034] running on GCE node
I20260924 04:10:07.396713 11703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:10:07.397538 11703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:10:07.398737 11703 hybrid_clock.cc:648] HybridClock initialized: now 1790223007398702 us; error 48 us; skew 500 ppm
Sep 24 04:10:07 dist-test-slave-00ld krb5kdc[11146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223007, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:10:07.402377 11703 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:10:07.403411 11703 webserver.cc:460] Webserver started at http://127.0.98.1:41681/ using document root <none> and password file <none>
I20260924 04:10:07.404103 11703 fs_manager.cc:359] Metadata directory not provided
I20260924 04:10:07.404188 11703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:10:07.404412 11703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 04:10:07 dist-test-slave-00ld krb5kdc[11146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223007, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 04:10:07.426690 11703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/minicluster-data/ts-0/data/instance:
uuid: "a2fa46813d4546778a50412344bf0802"
format_stamp: "Formatted at 2026-09-24 04:10:07 on dist-test-slave-00ld"
server_key: "e97923f5963902520b6dd7ba17e7cf3b"
server_key_iv: "5e764e9ae68805df0a4af8e64f9fed2a"
server_key_version: "kuduclusterkey@0"
I20260924 04:10:07.427445 11703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a2fa46813d4546778a50412344bf0802"
format_stamp: "Formatted at 2026-09-24 04:10:07 on dist-test-slave-00ld"
server_key: "e97923f5963902520b6dd7ba17e7cf3b"
server_key_iv: "5e764e9ae68805df0a4af8e64f9fed2a"
server_key_version: "kuduclusterkey@0"
I20260924 04:10:07.432469 11703 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260924 04:10:07.461199 11720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:10:07.463043 11703 fs_manager.cc:725] Time spent opening block manager: real 0.016s user 0.001s sys 0.000s
I20260924 04:10:07.463191 11703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/minicluster-data/ts-0/wal
uuid: "a2fa46813d4546778a50412344bf0802"
format_stamp: "Formatted at 2026-09-24 04:10:07 on dist-test-slave-00ld"
server_key: "e97923f5963902520b6dd7ba17e7cf3b"
server_key_iv: "5e764e9ae68805df0a4af8e64f9fed2a"
server_key_version: "kuduclusterkey@0"
I20260924 04:10:07.463346 11703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:10:07.492215 11703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:10:07.495869 11703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:10:07.496065 11703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:10:07.496778 11703 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:10:07.497786 11703 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:10:07.497854 11703 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:10:07.497925 11703 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:10:07.497953 11703 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:10:07.510799 11703 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34837
I20260924 04:10:07.511786 11703 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:10:07.513720 11835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34837 every 8 connection(s)
I20260924 04:10:07.518842 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 11703
I20260924 04:10:07.518970 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:10:07.519240 392 external_mini_cluster.cc:1428] Setting key c35309dfbc1328782147fd903dcde511
Sep 24 04:10:07 dist-test-slave-00ld krb5kdc[11146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223007, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:10:07.528460 11838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:10:07.514470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51911 (local address 127.0.98.62:37555)
0924 04:10:07.514761 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0924 04:10:07.514765 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:10:07.515321 (+ 556us) server_negotiation.cc:407] Connection header received
0924 04:10:07.516286 (+ 965us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:10:07.516290 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:10:07.516347 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:10:07.516461 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:10:07.517574 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:10:07.518126 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:10:07.519638 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:10:07.519948 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:10:07.520023 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:10:07.523565 (+ 3542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:10:07.523571 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:10:07.523575 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:10:07.523610 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:10:07.525684 (+ 2074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:10:07.526356 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:10:07.526363 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:10:07.526367 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:10:07.526461 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:10:07.526917 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:10:07.526921 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:10:07.526926 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:10:07.527190 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:10:07.527275 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 04:10:07.528101 (+ 826us) server_negotiation.cc:299] Negotiation successful
0924 04:10:07.528243 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":103,"threads_started":1}
I20260924 04:10:07.530416 11839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:10:07.514655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37555 (local address 127.0.98.1:51911)
0924 04:10:07.515167 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0924 04:10:07.515220 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 04:10:07.516068 (+ 848us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:10:07.516622 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:10:07.516630 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:10:07.516932 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:10:07.517436 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:10:07.517448 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:10:07.518261 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:10:07.518269 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:10:07.519493 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:10:07.519504 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:10:07.520219 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:10:07.520223 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:10:07.520402 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:10:07.521084 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:10:07.521104 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:10:07.523367 (+ 2263us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:10:07.525821 (+ 2454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:10:07.525829 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:10:07.525849 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:10:07.526224 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:10:07.526583 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:10:07.526587 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:10:07.526589 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:10:07.526793 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:10:07.527369 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:10:07.527379 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:10:07.527779 (+ 400us) client_negotiation.cc:769] Sending connection context
0924 04:10:07.528887 (+ 1108us) client_negotiation.cc:240] Negotiation successful
0924 04:10:07.529259 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":164,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 04:10:07.531833 11836 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37555
I20260924 04:10:07.532083 11836 heartbeater.cc:464] Registering TS with master...
I20260924 04:10:07.532742 11836 heartbeater.cc:511] Master 127.0.98.62:37555 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 04:10:07.534258 11622 ts_manager.cc:194] Registered new tserver with Master: a2fa46813d4546778a50412344bf0802 (127.0.98.1:34837)
I20260924 04:10:07.535815 11622 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:51911
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:10:07.552451 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:54703/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37555
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 04:10:07 dist-test-slave-00ld krb5kdc[11146](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790223007, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260924 04:10:07.689270 11842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:10:07.689545 11842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:10:07.689605 11842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:10:07.693850 11842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:10:07.693934 11842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:10:07.694039 11842 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:10:07.699476 11842 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:54703/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37555
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11842
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:10:07.700531 11842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:10:07.701576 11842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:10:07.711974 11849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:10:07.712373 11851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:10:07.712558 11848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:10:07.713708 11842 server_base.cc:1034] running on GCE node
I20260924 04:10:07.714143 11842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:10:07.714864 11842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:10:07.716045 11842 hybrid_clock.cc:648] HybridClock initialized: now 1790223007716016 us; error 50 us; skew 500 ppm
Sep 24 04:10:07 dist-test-slave-00ld krb5kdc[11146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223007, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:10:07.719655 11842 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:10:07.720736 11842 webserver.cc:460] Webserver started at http://127.0.98.2:38623/ using document root <none> and password file <none>
I20260924 04:10:07.721421 11842 fs_manager.cc:359] Metadata directory not provided
I20260924 04:10:07.721504 11842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:10:07.721717 11842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 04:10:07 dist-test-slave-00ld krb5kdc[11146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223007, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 04:10:07.739123 11842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/minicluster-data/ts-1/data/instance:
uuid: "8be5acc1dd5343128c6e940f6ac41e75"
format_stamp: "Formatted at 2026-09-24 04:10:07 on dist-test-slave-00ld"
server_key: "ac2ba6780de4d3cd7b318637b8f2db52"
server_key_iv: "4d75bc0a5283037b0b511963734e2072"
server_key_version: "kuduclusterkey@0"
I20260924 04:10:07.739791 11842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8be5acc1dd5343128c6e940f6ac41e75"
format_stamp: "Formatted at 2026-09-24 04:10:07 on dist-test-slave-00ld"
server_key: "ac2ba6780de4d3cd7b318637b8f2db52"
server_key_iv: "4d75bc0a5283037b0b511963734e2072"
server_key_version: "kuduclusterkey@0"
I20260924 04:10:07.745432 11842 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 04:10:07.761206 11858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:10:07.762944 11842 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260924 04:10:07.763072 11842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/minicluster-data/ts-1/wal
uuid: "8be5acc1dd5343128c6e940f6ac41e75"
format_stamp: "Formatted at 2026-09-24 04:10:07 on dist-test-slave-00ld"
server_key: "ac2ba6780de4d3cd7b318637b8f2db52"
server_key_iv: "4d75bc0a5283037b0b511963734e2072"
server_key_version: "kuduclusterkey@0"
I20260924 04:10:07.763199 11842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:10:07.793919 11842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:10:07.797686 11842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:10:07.797936 11842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:10:07.798625 11842 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:10:07.799659 11842 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:10:07.799724 11842 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:10:07.799789 11842 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:10:07.799825 11842 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:10:07.809954 11842 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39873
I20260924 04:10:07.811061 11842 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:10:07.811762 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 11842
I20260924 04:10:07.811880 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:10:07.812191 392 external_mini_cluster.cc:1428] Setting key 86018c5227cef9e7511bac1d92d8f178
I20260924 04:10:07.817221 11971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39873 every 8 connection(s)
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Sep 24 04:10:07 dist-test-slave-00ld krb5kdc[11146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223007, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:10:07.838091 11838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:10:07.820701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33545 (local address 127.0.98.62:37555)
0924 04:10:07.820839 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 04:10:07.820844 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:10:07.821529 (+ 685us) server_negotiation.cc:407] Connection header received
0924 04:10:07.822747 (+ 1218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:10:07.822751 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:10:07.822804 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:10:07.822860 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:10:07.824460 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:10:07.825315 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:10:07.827148 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:10:07.827491 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:10:07.827578 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:10:07.832316 (+ 4738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:10:07.832320 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:10:07.832323 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:10:07.832353 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:10:07.834138 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:10:07.835653 (+ 1515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:10:07.835656 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:10:07.835658 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:10:07.835746 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:10:07.836773 (+ 1027us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:10:07.836777 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:10:07.836779 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:10:07.837003 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:10:07.837073 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 04:10:07.837789 (+ 716us) server_negotiation.cc:299] Negotiation successful
0924 04:10:07.837921 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 04:10:07.839712 11975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:10:07.819537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37555 (local address 127.0.98.2:33545)
0924 04:10:07.821379 (+ 1842us) negotiation.cc:107] Waiting for socket to connect
0924 04:10:07.821434 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 04:10:07.822485 (+ 1051us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:10:07.823251 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:10:07.823259 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:10:07.823607 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:10:07.824287 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:10:07.824307 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:10:07.825565 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:10:07.825568 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:10:07.827007 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:10:07.827017 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:10:07.827786 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:10:07.827790 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:10:07.827979 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:10:07.828731 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:10:07.828752 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:10:07.832111 (+ 3359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:10:07.834290 (+ 2179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:10:07.834296 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:10:07.834312 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:10:07.834665 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:10:07.835861 (+ 1196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:10:07.835865 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:10:07.835867 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:10:07.836040 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:10:07.837154 (+ 1114us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:10:07.837160 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:10:07.837509 (+ 349us) client_negotiation.cc:769] Sending connection context
0924 04:10:07.838331 (+ 822us) client_negotiation.cc:240] Negotiation successful
0924 04:10:07.838691 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1610,"thread_start_us":148,"threads_started":1}
I20260924 04:10:07.841109 11972 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37555
I20260924 04:10:07.841387 11972 heartbeater.cc:464] Registering TS with master...
I20260924 04:10:07.842051 11972 heartbeater.cc:511] Master 127.0.98.62:37555 requested a full tablet report, sending...
I20260924 04:10:07.843212 11622 ts_manager.cc:194] Registered new tserver with Master: 8be5acc1dd5343128c6e940f6ac41e75 (127.0.98.2:39873)
I20260924 04:10:07.844040 11622 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33545
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:10:07.856755 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:54703/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37555
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:10:07.998528 11977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:10:07.998816 11977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:10:07.998898 11977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:10:08.003302 11977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:10:08.003397 11977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:10:08.003512 11977 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:10:08.010864 11977 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:54703/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37555
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:10:08.012007 11977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:10:08.013113 11977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:10:08.028193 11983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:10:08.028656 11982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:10:08.028929 11985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:10:08.030066 11977 server_base.cc:1034] running on GCE node
I20260924 04:10:08.030521 11977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:10:08.031333 11977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:10:08.032493 11977 hybrid_clock.cc:648] HybridClock initialized: now 1790223008032426 us; error 80 us; skew 500 ppm
Sep 24 04:10:08 dist-test-slave-00ld krb5kdc[11146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223008, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:10:08.035934 11977 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:10:08.036988 11977 webserver.cc:460] Webserver started at http://127.0.98.3:42927/ using document root <none> and password file <none>
I20260924 04:10:08.037662 11977 fs_manager.cc:359] Metadata directory not provided
I20260924 04:10:08.037752 11977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:10:08.037972 11977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 04:10:08 dist-test-slave-00ld krb5kdc[11146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223008, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 24 04:10:08 dist-test-slave-00ld krb5kdc[11146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223007, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 04:10:08.070883 11977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/minicluster-data/ts-2/data/instance:
uuid: "71ca5d429f624445819441d6e1a1cfee"
format_stamp: "Formatted at 2026-09-24 04:10:08 on dist-test-slave-00ld"
server_key: "57c3f04eeb81937b866c4f2320118c8b"
server_key_iv: "ac8fcef45db36d9f5a276b8e0efbed9c"
server_key_version: "kuduclusterkey@0"
I20260924 04:10:08.071722 11977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/minicluster-data/ts-2/wal/instance:
uuid: "71ca5d429f624445819441d6e1a1cfee"
format_stamp: "Formatted at 2026-09-24 04:10:08 on dist-test-slave-00ld"
server_key: "57c3f04eeb81937b866c4f2320118c8b"
server_key_iv: "ac8fcef45db36d9f5a276b8e0efbed9c"
server_key_version: "kuduclusterkey@0"
I20260924 04:10:08.078866 11977 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260924 04:10:08.100426 11992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:10:08.102188 11977 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 04:10:08.102357 11977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/minicluster-data/ts-2/wal
uuid: "71ca5d429f624445819441d6e1a1cfee"
format_stamp: "Formatted at 2026-09-24 04:10:08 on dist-test-slave-00ld"
server_key: "57c3f04eeb81937b866c4f2320118c8b"
server_key_iv: "ac8fcef45db36d9f5a276b8e0efbed9c"
server_key_version: "kuduclusterkey@0"
I20260924 04:10:08.102509 11977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:10:08.132516 11977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:10:08.135669 11977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:10:08.135876 11977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:10:08.136513 11977 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:10:08.137495 11977 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:10:08.137567 11977 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:10:08.137632 11977 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:10:08.137684 11977 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:10:08.149078 11977 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40663
I20260924 04:10:08.149518 12105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40663 every 8 connection(s)
I20260924 04:10:08.150187 11977 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:10:08.150679 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 11977
I20260924 04:10:08.150799 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:10:08.151109 392 external_mini_cluster.cc:1428] Setting key 7de9da64c1abb951ac4665090a3ba6a1
Sep 24 04:10:08 dist-test-slave-00ld krb5kdc[11146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223008, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:10:08.165865 11838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:10:08.153373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46961 (local address 127.0.98.62:37555)
0924 04:10:08.153496 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 04:10:08.153501 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:10:08.154477 (+ 976us) server_negotiation.cc:407] Connection header received
0924 04:10:08.155413 (+ 936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:10:08.155417 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:10:08.155466 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:10:08.155566 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:10:08.156847 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:10:08.157614 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:10:08.158924 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:10:08.159228 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:10:08.159341 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:10:08.161894 (+ 2553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:10:08.161898 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:10:08.161901 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:10:08.161930 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:10:08.163528 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:10:08.164119 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:10:08.164123 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:10:08.164125 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:10:08.164216 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:10:08.164616 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:10:08.164620 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:10:08.164621 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:10:08.164844 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:10:08.164959 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 04:10:08.165584 (+ 625us) server_negotiation.cc:299] Negotiation successful
0924 04:10:08.165706 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 04:10:08.166671 12108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:10:08.153850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37555 (local address 127.0.98.3:46961)
0924 04:10:08.154311 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0924 04:10:08.154365 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 04:10:08.155223 (+ 858us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:10:08.155803 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:10:08.155811 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:10:08.156078 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:10:08.156577 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:10:08.156643 (+ 66us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:10:08.157780 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:10:08.157785 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:10:08.158803 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:10:08.158813 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:10:08.159414 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:10:08.159417 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:10:08.159593 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:10:08.160206 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:10:08.160221 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:10:08.161729 (+ 1508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:10:08.163647 (+ 1918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:10:08.163653 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:10:08.163669 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:10:08.164013 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:10:08.164329 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:10:08.164332 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:10:08.164334 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:10:08.164517 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:10:08.165046 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:10:08.165051 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:10:08.165353 (+ 302us) client_negotiation.cc:769] Sending connection context
0924 04:10:08.165544 (+ 191us) client_negotiation.cc:240] Negotiation successful
0924 04:10:08.165863 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":139,"threads_started":1}
I20260924 04:10:08.167793 12106 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37555
I20260924 04:10:08.168046 12106 heartbeater.cc:464] Registering TS with master...
I20260924 04:10:08.168614 12106 heartbeater.cc:511] Master 127.0.98.62:37555 requested a full tablet report, sending...
I20260924 04:10:08.169803 11622 ts_manager.cc:194] Registered new tserver with Master: 71ca5d429f624445819441d6e1a1cfee (127.0.98.3:40663)
I20260924 04:10:08.170493 11622 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:46961
I20260924 04:10:08.177428 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:10:08.187575 11838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:10:08.179548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43654 (local address 127.0.98.62:37555)
0924 04:10:08.179691 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 04:10:08.179704 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 04:10:08.179836 (+ 132us) server_negotiation.cc:407] Connection header received
0924 04:10:08.179923 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:10:08.179928 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:10:08.180004 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:10:08.180088 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:10:08.180926 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:10:08.181663 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:10:08.182715 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:10:08.182949 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:10:08.183060 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:10:08.184463 (+ 1403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:10:08.184466 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:10:08.184469 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:10:08.184494 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:10:08.185969 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:10:08.186392 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:10:08.186397 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:10:08.186398 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:10:08.186482 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:10:08.186751 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:10:08.186756 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:10:08.186757 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:10:08.186927 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:10:08.187061 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0924 04:10:08.187300 (+ 239us) server_negotiation.cc:299] Negotiation successful
0924 04:10:08.187408 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 04:10:08.191582 11622 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43654:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:10:08.194948 11622 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:10:08.211141 12121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:10:08.205421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40663 (local address 127.0.0.1:48680)
0924 04:10:08.206342 (+ 921us) negotiation.cc:107] Waiting for socket to connect
0924 04:10:08.206355 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:10:08.206440 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:10:08.207208 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:10:08.207213 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:10:08.207236 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:10:08.207365 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:10:08.207370 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:10:08.208524 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:10:08.208530 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:10:08.209813 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:10:08.209822 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:10:08.210609 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:10:08.210613 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:10:08.210735 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:10:08.210773 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 04:10:08.210881 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 04:10:08.210998 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":858,"thread_start_us":81,"threads_started":1}
I20260924 04:10:08.211153 12122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:10:08.205818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48680 (local address 127.0.98.3:40663)
0924 04:10:08.206201 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0924 04:10:08.206215 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 04:10:08.206537 (+ 322us) server_negotiation.cc:407] Connection header received
0924 04:10:08.206624 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:10:08.206630 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:10:08.206873 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:10:08.207033 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:10:08.207480 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:10:08.208418 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:10:08.209955 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:10:08.210491 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:10:08.210611 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:10:08.210762 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 04:10:08.210921 (+ 159us) server_negotiation.cc:299] Negotiation successful
0924 04:10:08.211009 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":128,"threads_started":1}
I20260924 04:10:08.214214 12039 tablet_service.cc:1467] Processing CreateTablet for tablet e141e70eed9c44c38be9b1f9c619ca59 (DEFAULT_TABLE table=test-table [id=862fdb46429740828d59325abc41ab76]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:10:08.215433 12039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e141e70eed9c44c38be9b1f9c619ca59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:10:08.222661 12120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:10:08.205772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39873 (local address 127.0.0.1:45666)
0924 04:10:08.206972 (+ 1200us) negotiation.cc:107] Waiting for socket to connect
0924 04:10:08.206987 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:10:08.207058 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:10:08.217386 (+ 10328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:10:08.217390 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:10:08.217409 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:10:08.217506 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:10:08.217512 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:10:08.218913 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:10:08.218916 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:10:08.220139 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:10:08.220153 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:10:08.222239 (+ 2086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:10:08.222242 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:10:08.222352 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:10:08.222377 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 04:10:08.222448 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 04:10:08.222514 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1087,"mutex_wait_us":77,"thread_start_us":83,"threads_started":1}
I20260924 04:10:08.223249 12123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:10:08.205976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45666 (local address 127.0.98.2:39873)
0924 04:10:08.216895 (+ 10919us) server_negotiation.cc:206] Beginning negotiation
0924 04:10:08.216906 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 04:10:08.216931 (+ 25us) server_negotiation.cc:407] Connection header received
0924 04:10:08.217003 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:10:08.217010 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:10:08.217229 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:10:08.217391 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:10:08.217647 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:10:08.218775 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:10:08.221034 (+ 2259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:10:08.221920 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:10:08.222026 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:10:08.222166 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 04:10:08.222981 (+ 815us) server_negotiation.cc:299] Negotiation successful
0924 04:10:08.223087 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10810,"thread_start_us":143,"threads_started":1}
I20260924 04:10:08.224009 12124 tablet_bootstrap.cc:492] T e141e70eed9c44c38be9b1f9c619ca59 P 71ca5d429f624445819441d6e1a1cfee: Bootstrap starting.
I20260924 04:10:08.226794 12124 tablet_bootstrap.cc:654] T e141e70eed9c44c38be9b1f9c619ca59 P 71ca5d429f624445819441d6e1a1cfee: Neither blocks nor log segments found. Creating new log.
I20260924 04:10:08.226523 11906 tablet_service.cc:1467] Processing CreateTablet for tablet e141e70eed9c44c38be9b1f9c619ca59 (DEFAULT_TABLE table=test-table [id=862fdb46429740828d59325abc41ab76]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:10:08.227681 12124 log.cc:824] T e141e70eed9c44c38be9b1f9c619ca59 P 71ca5d429f624445819441d6e1a1cfee: Log is configured to *not* fsync() on all Append() calls
I20260924 04:10:08.228343 11906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e141e70eed9c44c38be9b1f9c619ca59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:10:08.229202 12118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:10:08.204168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34837 (local address 127.0.0.1:49240)
0924 04:10:08.204502 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0924 04:10:08.204546 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 04:10:08.204660 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:10:08.218096 (+ 13436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:10:08.218099 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:10:08.218115 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:10:08.218220 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:10:08.218225 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:10:08.219747 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:10:08.219750 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:10:08.221030 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:10:08.221040 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:10:08.228789 (+ 7749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:10:08.228793 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:10:08.228897 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:10:08.228924 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 04:10:08.228979 (+ 55us) client_negotiation.cc:240] Negotiation successful
0924 04:10:08.229050 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"mutex_wait_us":44,"thread_start_us":103,"threads_started":1}
I20260924 04:10:08.229955 12119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:10:08.205000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49240 (local address 127.0.98.1:34837)
0924 04:10:08.217670 (+ 12670us) server_negotiation.cc:206] Beginning negotiation
0924 04:10:08.217677 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 04:10:08.217700 (+ 23us) server_negotiation.cc:407] Connection header received
0924 04:10:08.217770 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:10:08.217775 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:10:08.217949 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:10:08.218369 (+ 420us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:10:08.218507 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:10:08.219610 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:10:08.221304 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:10:08.222204 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:10:08.229485 (+ 7281us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:10:08.229646 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0924 04:10:08.229705 (+ 59us) server_negotiation.cc:299] Negotiation successful
0924 04:10:08.229804 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12565,"thread_start_us":89,"threads_started":1}
I20260924 04:10:08.233237 11766 tablet_service.cc:1467] Processing CreateTablet for tablet e141e70eed9c44c38be9b1f9c619ca59 (DEFAULT_TABLE table=test-table [id=862fdb46429740828d59325abc41ab76]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:10:08.234287 11766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e141e70eed9c44c38be9b1f9c619ca59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:10:08.240866 12127 tablet_bootstrap.cc:492] T e141e70eed9c44c38be9b1f9c619ca59 P a2fa46813d4546778a50412344bf0802: Bootstrap starting.
I20260924 04:10:08.242579 12124 tablet_bootstrap.cc:492] T e141e70eed9c44c38be9b1f9c619ca59 P 71ca5d429f624445819441d6e1a1cfee: No bootstrap required, opened a new log
I20260924 04:10:08.242817 12124 ts_tablet_manager.cc:1397] T e141e70eed9c44c38be9b1f9c619ca59 P 71ca5d429f624445819441d6e1a1cfee: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.000s
I20260924 04:10:08.243042 12127 tablet_bootstrap.cc:654] T e141e70eed9c44c38be9b1f9c619ca59 P a2fa46813d4546778a50412344bf0802: Neither blocks nor log segments found. Creating new log.
I20260924 04:10:08.243862 12127 log.cc:824] T e141e70eed9c44c38be9b1f9c619ca59 P a2fa46813d4546778a50412344bf0802: Log is configured to *not* fsync() on all Append() calls
I20260924 04:10:08.244647 12126 tablet_bootstrap.cc:492] T e141e70eed9c44c38be9b1f9c619ca59 P 8be5acc1dd5343128c6e940f6ac41e75: Bootstrap starting.
I20260924 04:10:08.245814 12127 tablet_bootstrap.cc:492] T e141e70eed9c44c38be9b1f9c619ca59 P a2fa46813d4546778a50412344bf0802: No bootstrap required, opened a new log
I20260924 04:10:08.246021 12127 ts_tablet_manager.cc:1397] T e141e70eed9c44c38be9b1f9c619ca59 P a2fa46813d4546778a50412344bf0802: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260924 04:10:08.246944 12124 raft_consensus.cc:348] T e141e70eed9c44c38be9b1f9c619ca59 P 71ca5d429f624445819441d6e1a1cfee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2fa46813d4546778a50412344bf0802" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34837 } } peers { permanent_uuid: "71ca5d429f624445819441d6e1a1cfee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40663 } } peers { permanent_uuid: "8be5acc1dd5343128c6e940f6ac41e75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39873 } }
I20260924 04:10:08.248611 12126 tablet_bootstrap.cc:654] T e141e70eed9c44c38be9b1f9c619ca59 P 8be5acc1dd5343128c6e940f6ac41e75: Neither blocks nor log segments found. Creating new log.
I20260924 04:10:08.249585 12126 log.cc:824] T e141e70eed9c44c38be9b1f9c619ca59 P 8be5acc1dd5343128c6e940f6ac41e75: Log is configured to *not* fsync() on all Append() calls
I20260924 04:10:08.250418 12127 raft_consensus.cc:348] T e141e70eed9c44c38be9b1f9c619ca59 P a2fa46813d4546778a50412344bf0802 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2fa46813d4546778a50412344bf0802" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34837 } } peers { permanent_uuid: "71ca5d429f624445819441d6e1a1cfee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40663 } } peers { permanent_uuid: "8be5acc1dd5343128c6e940f6ac41e75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39873 } }
I20260924 04:10:08.250731 12127 raft_consensus.cc:374] T e141e70eed9c44c38be9b1f9c619ca59 P a2fa46813d4546778a50412344bf0802 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:10:08.250829 12127 raft_consensus.cc:729] T e141e70eed9c44c38be9b1f9c619ca59 P a2fa46813d4546778a50412344bf0802 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2fa46813d4546778a50412344bf0802, State: Initialized, Role: FOLLOWER
I20260924 04:10:08.251389 12127 consensus_queue.cc:260] T e141e70eed9c44c38be9b1f9c619ca59 P a2fa46813d4546778a50412344bf0802 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2fa46813d4546778a50412344bf0802" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34837 } } peers { permanent_uuid: "71ca5d429f624445819441d6e1a1cfee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40663 } } peers { permanent_uuid: "8be5acc1dd5343128c6e940f6ac41e75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39873 } }
I20260924 04:10:08.251638 12126 tablet_bootstrap.cc:492] T e141e70eed9c44c38be9b1f9c619ca59 P 8be5acc1dd5343128c6e940f6ac41e75: No bootstrap required, opened a new log
I20260924 04:10:08.251820 12126 ts_tablet_manager.cc:1397] T e141e70eed9c44c38be9b1f9c619ca59 P 8be5acc1dd5343128c6e940f6ac41e75: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.006s
I20260924 04:10:08.252509 12127 ts_tablet_manager.cc:1428] T e141e70eed9c44c38be9b1f9c619ca59 P a2fa46813d4546778a50412344bf0802: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260924 04:10:08.252889 11836 heartbeater.cc:503] Master 127.0.98.62:37555 was elected leader, sending a full tablet report...
I20260924 04:10:08.255890 12124 raft_consensus.cc:374] T e141e70eed9c44c38be9b1f9c619ca59 P 71ca5d429f624445819441d6e1a1cfee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:10:08.255976 12124 raft_consensus.cc:729] T e141e70eed9c44c38be9b1f9c619ca59 P 71ca5d429f624445819441d6e1a1cfee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71ca5d429f624445819441d6e1a1cfee, State: Initialized, Role: FOLLOWER
I20260924 04:10:08.256500 12124 consensus_queue.cc:260] T e141e70eed9c44c38be9b1f9c619ca59 P 71ca5d429f624445819441d6e1a1cfee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2fa46813d4546778a50412344bf0802" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34837 } } peers { permanent_uuid: "71ca5d429f624445819441d6e1a1cfee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40663 } } peers { permanent_uuid: "8be5acc1dd5343128c6e940f6ac41e75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39873 } }
I20260924 04:10:08.257091 12126 raft_consensus.cc:348] T e141e70eed9c44c38be9b1f9c619ca59 P 8be5acc1dd5343128c6e940f6ac41e75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2fa46813d4546778a50412344bf0802" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34837 } } peers { permanent_uuid: "71ca5d429f624445819441d6e1a1cfee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40663 } } peers { permanent_uuid: "8be5acc1dd5343128c6e940f6ac41e75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39873 } }
I20260924 04:10:08.257503 12126 raft_consensus.cc:374] T e141e70eed9c44c38be9b1f9c619ca59 P 8be5acc1dd5343128c6e940f6ac41e75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:10:08.257607 12126 raft_consensus.cc:729] T e141e70eed9c44c38be9b1f9c619ca59 P 8be5acc1dd5343128c6e940f6ac41e75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8be5acc1dd5343128c6e940f6ac41e75, State: Initialized, Role: FOLLOWER
I20260924 04:10:08.257663 12124 ts_tablet_manager.cc:1428] T e141e70eed9c44c38be9b1f9c619ca59 P 71ca5d429f624445819441d6e1a1cfee: Time spent starting tablet: real 0.015s user 0.003s sys 0.004s
I20260924 04:10:08.258095 12106 heartbeater.cc:503] Master 127.0.98.62:37555 was elected leader, sending a full tablet report...
I20260924 04:10:08.258124 12126 consensus_queue.cc:260] T e141e70eed9c44c38be9b1f9c619ca59 P 8be5acc1dd5343128c6e940f6ac41e75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2fa46813d4546778a50412344bf0802" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34837 } } peers { permanent_uuid: "71ca5d429f624445819441d6e1a1cfee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40663 } } peers { permanent_uuid: "8be5acc1dd5343128c6e940f6ac41e75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39873 } }
I20260924 04:10:08.258985 12126 ts_tablet_manager.cc:1428] T e141e70eed9c44c38be9b1f9c619ca59 P 8be5acc1dd5343128c6e940f6ac41e75: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260924 04:10:08.259359 11972 heartbeater.cc:503] Master 127.0.98.62:37555 was elected leader, sending a full tablet report...
W20260924 04:10:08.270452 11837 tablet.cc:2369] T e141e70eed9c44c38be9b1f9c619ca59 P a2fa46813d4546778a50412344bf0802: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 04:10:08.321576 11973 tablet.cc:2369] T e141e70eed9c44c38be9b1f9c619ca59 P 8be5acc1dd5343128c6e940f6ac41e75: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 04:10:08 dist-test-slave-00ld krb5kdc[11146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223007, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260924 04:10:08.403016 12107 tablet.cc:2369] T e141e70eed9c44c38be9b1f9c619ca59 P 71ca5d429f624445819441d6e1a1cfee: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:10:08.417374 12130 raft_consensus.cc:482] T e141e70eed9c44c38be9b1f9c619ca59 P a2fa46813d4546778a50412344bf0802 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:10:08.417610 12130 raft_consensus.cc:504] T e141e70eed9c44c38be9b1f9c619ca59 P a2fa46813d4546778a50412344bf0802 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2fa46813d4546778a50412344bf0802" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34837 } } peers { permanent_uuid: "71ca5d429f624445819441d6e1a1cfee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40663 } } peers { permanent_uuid: "8be5acc1dd5343128c6e940f6ac41e75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39873 } }
I20260924 04:10:08.418967 12130 leader_election.cc:288] T e141e70eed9c44c38be9b1f9c619ca59 P a2fa46813d4546778a50412344bf0802 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71ca5d429f624445819441d6e1a1cfee (127.0.98.3:40663), 8be5acc1dd5343128c6e940f6ac41e75 (127.0.98.2:39873)
I20260924 04:10:08.426668 12123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:10:08.419745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37829 (local address 127.0.98.2:39873)
0924 04:10:08.420013 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0924 04:10:08.420018 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:10:08.422017 (+ 1999us) server_negotiation.cc:407] Connection header received
0924 04:10:08.422253 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:10:08.422257 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:10:08.422309 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:10:08.422360 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:10:08.422822 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:10:08.423613 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:10:08.425266 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:10:08.425924 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:10:08.426011 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:10:08.426081 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 04:10:08.426448 (+ 367us) server_negotiation.cc:299] Negotiation successful
0924 04:10:08.426530 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179}
I20260924 04:10:08.427291 12136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:10:08.419235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39873 (local address 127.0.98.1:37829)
0924 04:10:08.421909 (+ 2674us) negotiation.cc:107] Waiting for socket to connect
0924 04:10:08.421925 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:10:08.422104 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:10:08.422511 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:10:08.422514 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:10:08.422530 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:10:08.422671 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:10:08.422678 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:10:08.423776 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:10:08.423779 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:10:08.425116 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:10:08.425127 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:10:08.426195 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:10:08.426198 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:10:08.426299 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:10:08.426324 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 04:10:08.427040 (+ 716us) client_negotiation.cc:240] Negotiation successful
0924 04:10:08.427126 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2586,"thread_start_us":126,"threads_started":1}
I20260924 04:10:08.428177 11916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e141e70eed9c44c38be9b1f9c619ca59" candidate_uuid: "a2fa46813d4546778a50412344bf0802" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8be5acc1dd5343128c6e940f6ac41e75" is_pre_election: true
I20260924 04:10:08.428565 11916 raft_consensus.cc:2454] T e141e70eed9c44c38be9b1f9c619ca59 P 8be5acc1dd5343128c6e940f6ac41e75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2fa46813d4546778a50412344bf0802 in term 0.
I20260924 04:10:08.429476 11722 leader_election.cc:302] T e141e70eed9c44c38be9b1f9c619ca59 P a2fa46813d4546778a50412344bf0802 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8be5acc1dd5343128c6e940f6ac41e75, a2fa46813d4546778a50412344bf0802; no voters:
I20260924 04:10:08.429806 12130 raft_consensus.cc:2781] T e141e70eed9c44c38be9b1f9c619ca59 P a2fa46813d4546778a50412344bf0802 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:10:08.429890 12130 raft_consensus.cc:482] T e141e70eed9c44c38be9b1f9c619ca59 P a2fa46813d4546778a50412344bf0802 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:10:08.429937 12130 raft_consensus.cc:3037] T e141e70eed9c44c38be9b1f9c619ca59 P a2fa46813d4546778a50412344bf0802 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:10:08.431285 12130 raft_consensus.cc:504] T e141e70eed9c44c38be9b1f9c619ca59 P a2fa46813d4546778a50412344bf0802 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2fa46813d4546778a50412344bf0802" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34837 } } peers { permanent_uuid: "71ca5d429f624445819441d6e1a1cfee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40663 } } peers { permanent_uuid: "8be5acc1dd5343128c6e940f6ac41e75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39873 } }
I20260924 04:10:08.431694 12130 leader_election.cc:288] T e141e70eed9c44c38be9b1f9c619ca59 P a2fa46813d4546778a50412344bf0802 [CANDIDATE]: Term 1 election: Requested vote from peers 71ca5d429f624445819441d6e1a1cfee (127.0.98.3:40663), 8be5acc1dd5343128c6e940f6ac41e75 (127.0.98.2:39873)
I20260924 04:10:08.432363 11916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e141e70eed9c44c38be9b1f9c619ca59" candidate_uuid: "a2fa46813d4546778a50412344bf0802" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8be5acc1dd5343128c6e940f6ac41e75"
I20260924 04:10:08.432482 11916 raft_consensus.cc:3037] T e141e70eed9c44c38be9b1f9c619ca59 P 8be5acc1dd5343128c6e940f6ac41e75 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:10:08.433751 11916 raft_consensus.cc:2454] T e141e70eed9c44c38be9b1f9c619ca59 P 8be5acc1dd5343128c6e940f6ac41e75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2fa46813d4546778a50412344bf0802 in term 1.
I20260924 04:10:08.434130 11722 leader_election.cc:302] T e141e70eed9c44c38be9b1f9c619ca59 P a2fa46813d4546778a50412344bf0802 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8be5acc1dd5343128c6e940f6ac41e75, a2fa46813d4546778a50412344bf0802; no voters:
I20260924 04:10:08.434356 12130 raft_consensus.cc:2781] T e141e70eed9c44c38be9b1f9c619ca59 P a2fa46813d4546778a50412344bf0802 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:10:08.434563 12130 raft_consensus.cc:686] T e141e70eed9c44c38be9b1f9c619ca59 P a2fa46813d4546778a50412344bf0802 [term 1 LEADER]: Becoming Leader. State: Replica: a2fa46813d4546778a50412344bf0802, State: Running, Role: LEADER
I20260924 04:10:08.434942 12130 consensus_queue.cc:237] T e141e70eed9c44c38be9b1f9c619ca59 P a2fa46813d4546778a50412344bf0802 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2fa46813d4546778a50412344bf0802" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34837 } } peers { permanent_uuid: "71ca5d429f624445819441d6e1a1cfee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40663 } } peers { permanent_uuid: "8be5acc1dd5343128c6e940f6ac41e75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39873 } }
I20260924 04:10:08.438524 11621 catalog_manager.cc:5495] T e141e70eed9c44c38be9b1f9c619ca59 P a2fa46813d4546778a50412344bf0802 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2fa46813d4546778a50412344bf0802 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "a2fa46813d4546778a50412344bf0802" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2fa46813d4546778a50412344bf0802" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "71ca5d429f624445819441d6e1a1cfee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8be5acc1dd5343128c6e940f6ac41e75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39873 } health_report { overall_health: UNKNOWN } } }
I20260924 04:10:08.443123 12122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:10:08.419874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51425 (local address 127.0.98.3:40663)
0924 04:10:08.420100 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0924 04:10:08.420104 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:10:08.427154 (+ 7050us) server_negotiation.cc:407] Connection header received
0924 04:10:08.427381 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:10:08.427385 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:10:08.427443 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:10:08.427494 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:10:08.437400 (+ 9906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:10:08.438080 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:10:08.439562 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:10:08.440165 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:10:08.440249 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:10:08.440332 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 04:10:08.442890 (+ 2558us) server_negotiation.cc:299] Negotiation successful
0924 04:10:08.442985 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159}
I20260924 04:10:08.443766 12135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:10:08.419235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40663 (local address 127.0.98.1:51425)
0924 04:10:08.427052 (+ 7817us) negotiation.cc:107] Waiting for socket to connect
0924 04:10:08.427069 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 04:10:08.427232 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:10:08.437072 (+ 9840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:10:08.437075 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:10:08.437091 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:10:08.437221 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:10:08.437228 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:10:08.438252 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:10:08.438255 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:10:08.439417 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:10:08.439427 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:10:08.442625 (+ 3198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:10:08.442630 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:10:08.442730 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:10:08.442760 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 04:10:08.443516 (+ 756us) client_negotiation.cc:240] Negotiation successful
0924 04:10:08.443607 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7741,"mutex_wait_us":877,"thread_start_us":100,"threads_started":1}
I20260924 04:10:08.444530 12060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e141e70eed9c44c38be9b1f9c619ca59" candidate_uuid: "a2fa46813d4546778a50412344bf0802" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71ca5d429f624445819441d6e1a1cfee"
I20260924 04:10:08.444561 12058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e141e70eed9c44c38be9b1f9c619ca59" candidate_uuid: "a2fa46813d4546778a50412344bf0802" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71ca5d429f624445819441d6e1a1cfee" is_pre_election: true
I20260924 04:10:08.444805 12060 raft_consensus.cc:3037] T e141e70eed9c44c38be9b1f9c619ca59 P 71ca5d429f624445819441d6e1a1cfee [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:10:08.446266 12060 raft_consensus.cc:2454] T e141e70eed9c44c38be9b1f9c619ca59 P 71ca5d429f624445819441d6e1a1cfee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2fa46813d4546778a50412344bf0802 in term 1.
I20260924 04:10:08.509753 12119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:10:08.504308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49242 (local address 127.0.98.1:34837)
0924 04:10:08.504467 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 04:10:08.504471 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:10:08.504564 (+ 93us) server_negotiation.cc:407] Connection header received
0924 04:10:08.504786 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:10:08.504789 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:10:08.504848 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:10:08.504972 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:10:08.505246 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:10:08.506040 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:10:08.507410 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:10:08.507760 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:10:08.507862 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:10:08.508283 (+ 421us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:10:08.508967 (+ 684us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:10:08.509083 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 04:10:08.509518 (+ 435us) server_negotiation.cc:299] Negotiation successful
0924 04:10:08.509581 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 04:10:08.517892 11916 raft_consensus.cc:1261] T e141e70eed9c44c38be9b1f9c619ca59 P 8be5acc1dd5343128c6e940f6ac41e75 [term 1 FOLLOWER]: Refusing update from remote peer a2fa46813d4546778a50412344bf0802: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:10:08.518702 12060 raft_consensus.cc:1261] T e141e70eed9c44c38be9b1f9c619ca59 P 71ca5d429f624445819441d6e1a1cfee [term 1 FOLLOWER]: Refusing update from remote peer a2fa46813d4546778a50412344bf0802: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:10:08.518680 12137 consensus_queue.cc:1035] T e141e70eed9c44c38be9b1f9c619ca59 P a2fa46813d4546778a50412344bf0802 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8be5acc1dd5343128c6e940f6ac41e75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:10:08.519445 12137 consensus_queue.cc:1035] T e141e70eed9c44c38be9b1f9c619ca59 P a2fa46813d4546778a50412344bf0802 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71ca5d429f624445819441d6e1a1cfee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:10:08.539255 12140 mvcc.cc:205] Tried to move back new op lower bound from 7332753442881417216 to 7332753442553282560. Current Snapshot: MvccSnapshot[applied={T|T < 7332753442881417216}]
I20260924 04:10:08.539769 12142 mvcc.cc:205] Tried to move back new op lower bound from 7332753442881417216 to 7332753442553282560. Current Snapshot: MvccSnapshot[applied={T|T < 7332753442881417216}]
I20260924 04:10:08.553427 11621 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43654:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:10:08.553611 11621 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43654:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:10:08.556362 11621 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3f488253ca6f40a9bb4747bbc564a018: Sending DeleteTablet for 3 replicas of tablet e141e70eed9c44c38be9b1f9c619ca59
I20260924 04:10:08.557443 12039 tablet_service.cc:1514] Processing DeleteTablet for tablet e141e70eed9c44c38be9b1f9c619ca59 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:10:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48680
I20260924 04:10:08.557469 11906 tablet_service.cc:1514] Processing DeleteTablet for tablet e141e70eed9c44c38be9b1f9c619ca59 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:10:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45666
I20260924 04:10:08.557924 12150 tablet_replica.cc:330] T e141e70eed9c44c38be9b1f9c619ca59 P 71ca5d429f624445819441d6e1a1cfee: stopping tablet replica
I20260924 04:10:08.558092 11766 tablet_service.cc:1514] Processing DeleteTablet for tablet e141e70eed9c44c38be9b1f9c619ca59 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:10:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49240
I20260924 04:10:08.558228 12150 raft_consensus.cc:2229] T e141e70eed9c44c38be9b1f9c619ca59 P 71ca5d429f624445819441d6e1a1cfee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:10:08.558554 12150 raft_consensus.cc:2258] T e141e70eed9c44c38be9b1f9c619ca59 P 71ca5d429f624445819441d6e1a1cfee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:10:08.559085 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 11703
I20260924 04:10:08.561972 12150 ts_tablet_manager.cc:1905] T e141e70eed9c44c38be9b1f9c619ca59 P 71ca5d429f624445819441d6e1a1cfee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:10:08.565521 12150 ts_tablet_manager.cc:1918] T e141e70eed9c44c38be9b1f9c619ca59 P 71ca5d429f624445819441d6e1a1cfee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:10:08.565713 12150 log.cc:1196] T e141e70eed9c44c38be9b1f9c619ca59 P 71ca5d429f624445819441d6e1a1cfee: Deleting WAL directory at /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790222898807720-392-0/minicluster-data/ts-2/wal/wals/e141e70eed9c44c38be9b1f9c619ca59
I20260924 04:10:08.566471 12150 ts_tablet_manager.cc:1939] T e141e70eed9c44c38be9b1f9c619ca59 P 71ca5d429f624445819441d6e1a1cfee: Deleting consensus metadata
I20260924 04:10:08.566784 12151 tablet_replica.cc:330] T e141e70eed9c44c38be9b1f9c619ca59 P 8be5acc1dd5343128c6e940f6ac41e75: stopping tablet replica
I20260924 04:10:08.567209 12151 raft_consensus.cc:2229] T e141e70eed9c44c38be9b1f9c619ca59 P 8be5acc1dd5343128c6e940f6ac41e75 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260924 04:10:08.568143 11606 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34837 (error 108)
I20260924 04:10:08.568641 12151 raft_consensus.cc:2258] T e141e70eed9c44c38be9b1f9c619ca59 P 8be5acc1dd5343128c6e940f6ac41e75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:10:08.569206 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 11842
I20260924 04:10:08.575253 11609 catalog_manager.cc:4841] TS 71ca5d429f624445819441d6e1a1cfee (127.0.98.3:40663): tablet e141e70eed9c44c38be9b1f9c619ca59 (table test-table [id=862fdb46429740828d59325abc41ab76]) successfully deleted
I20260924 04:10:08.576707 12120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:10:08.576360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34837 (local address 127.0.0.1:49258)
0924 04:10:08.576493 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 04:10:08.576578 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260924 04:10:08.576788 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 11977
W20260924 04:10:08.578835 11606 catalog_manager.cc:4584] TS a2fa46813d4546778a50412344bf0802 (127.0.98.1:34837): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e141e70eed9c44c38be9b1f9c619ca59: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34837: connect: Connection refused (error 111)
I20260924 04:10:08.579384 12120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:10:08.579137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39873 (local address 127.0.0.1:45672)
0924 04:10:08.579243 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0924 04:10:08.579300 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:39873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260924 04:10:08.584550 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 11590
I20260924 04:10:08.591486 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 04:10:08.926350 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 04:10:08.926477 392 mini_ranger.cc:66] Stopping Ranger...
I20260924 04:10:09.271404 392 mini_ranger.cc:68] Stopped Ranger
2026-09-24 04:10:09.271 UTC [11176] LOG: received smart shutdown request
2026-09-24 04:10:09.275 UTC [11176] LOG: background worker "logical replication launcher" (PID 11185) exited with exit code 1
2026-09-24 04:10:09.275 UTC [11180] LOG: shutting down
2026-09-24 04:10:09.449 UTC [11176] LOG: database system is shut down
2026-09-24T04:10:09Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (46818 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:10:09 dist-test-slave-00ld krb5kdc[12162](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:10:09 dist-test-slave-00ld krb5kdc[12162](info): set up 2 sockets
Sep 24 04:10:09 dist-test-slave-00ld krb5kdc[12162](info): commencing operation
krb5kdc: starting...
W20260924 04:10:10.612843 392 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/krb5kdc/test-user.keytab.
Sep 24 04:10:10 dist-test-slave-00ld krb5kdc[12162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223010, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:10:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:10:10Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260924 04:10:10.778970 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task1CVu5m/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/postgres -l logfile start
2026-09-24 04:10:13.267 UTC [12192] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 04:10:13.267 UTC [12192] LOG: listening on IPv4 address "127.0.98.20", port 57231
2026-09-24 04:10:13.270 UTC [12192] LOG: listening on Unix socket "/tmp/.s.PGSQL.57231"
2026-09-24 04:10:13.304 UTC [12195] LOG: database system was shut down at 2026-09-24 04:10:11 UTC
2026-09-24 04:10:13.308 UTC [12192] LOG: database system is ready to accept connections
I20260924 04:10:14.248799 392 mini_postgres.cc:96] Postgres bound to 57231
2026-09-24 04:10:14.255 UTC [12203] FATAL: database "slave" does not exist
127.0.98.20:57231 - accepting connections
I20260924 04:10:14.256261 392 mini_ranger.cc:161] Starting Ranger...
I20260924 04:10:14.272643 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 04:10:14.789198 392 mini_ranger.cc:87] Created ranger Postgres database
I20260924 04:10:14.789275 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ranger-admin
2026-09-24 04:10:15,064 [I] DB FLAVOR :POSTGRES
2026-09-24 04:10:15,064 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 04:10:15,064 [I] Checking connection..
2026-09-24 04:10:15,065 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1CVu5m/build/debug/bin/postgresql.jar:/tmp/dist-test-task1CVu5m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 04:10:15,321 [I] Checking connection passed.
2026-09-24 04:10:15,322 [I] --------- Verifying version history table ---------
2026-09-24 04:10:15,322 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1CVu5m/build/debug/bin/postgresql.jar:/tmp/dist-test-task1CVu5m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 04:10:15,586 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 04:10:15,587 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1CVu5m/build/debug/bin/postgresql.jar:/tmp/dist-test-task1CVu5m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 04:10:15,817 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 04:10:15,817 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 04:10:15,818 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1CVu5m/build/debug/bin/postgresql.jar:/tmp/dist-test-task1CVu5m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57231/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task1CVu5m/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 04:10:16.062 UTC [12267] WARNING: there is no transaction in progress
2026-09-24 04:10:16,068 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 04:10:16,069 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1CVu5m/build/debug/bin/postgresql.jar:/tmp/dist-test-task1CVu5m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 04:10:16,299 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 04:10:16,299 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 04:10:16,299 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1CVu5m/build/debug/bin/postgresql.jar:/tmp/dist-test-task1CVu5m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 04:10:16,561 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1CVu5m/build/debug/bin/postgresql.jar:/tmp/dist-test-task1CVu5m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 04:10:16,796 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1CVu5m/build/debug/bin/postgresql.jar:/tmp/dist-test-task1CVu5m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-00ld.c.gcp-upstream.internal','N') ;"
2026-09-24 04:10:17,017 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1CVu5m/build/debug/bin/postgresql.jar:/tmp/dist-test-task1CVu5m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 04:10:17,278 [I] Table x_portal_user does not exist in database ranger
2026-09-24 04:10:17,278 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1CVu5m/build/debug/bin/postgresql.jar:/tmp/dist-test-task1CVu5m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 04:10:17,536 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 04:10:17,537 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1CVu5m/build/debug/bin/postgresql.jar:/tmp/dist-test-task1CVu5m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 04:10:17,794 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1CVu5m/build/debug/bin/postgresql.jar:/tmp/dist-test-task1CVu5m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 04:10:18,050 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 04:10:18,050 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1CVu5m/build/debug/bin/postgresql.jar:/tmp/dist-test-task1CVu5m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57231/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task1CVu5m/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 04:10:19.012 UTC [12375] WARNING: there is no transaction in progress
2026-09-24 04:10:19.029 UTC [12375] WARNING: there is no transaction in progress
2026-09-24 04:10:19.047 UTC [12375] WARNING: there is no transaction in progress
2026-09-24 04:10:19.063 UTC [12375] WARNING: there is no transaction in progress
2026-09-24 04:10:19.082 UTC [12375] WARNING: there is no transaction in progress
2026-09-24 04:10:19.100 UTC [12375] WARNING: there is no transaction in progress
2026-09-24 04:10:19.161 UTC [12375] WARNING: there is no transaction in progress
2026-09-24 04:10:19.169 UTC [12375] WARNING: there is no transaction in progress
2026-09-24 04:10:19.184 UTC [12375] WARNING: there is no transaction in progress
2026-09-24 04:10:19.195 UTC [12375] WARNING: there is no transaction in progress
2026-09-24 04:10:19.205 UTC [12375] WARNING: there is no transaction in progress
2026-09-24 04:10:19.219 UTC [12375] WARNING: there is no transaction in progress
2026-09-24 04:10:19.228 UTC [12375] WARNING: there is no transaction in progress
2026-09-24 04:10:19.235 UTC [12375] WARNING: there is no transaction in progress
2026-09-24 04:10:19.242 UTC [12375] WARNING: there is no transaction in progress
2026-09-24 04:10:19.250 UTC [12375] WARNING: there is no transaction in progress
2026-09-24 04:10:19,915 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 04:10:19,916 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1CVu5m/build/debug/bin/postgresql.jar:/tmp/dist-test-task1CVu5m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 04:10:20,170 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 04:10:20,170 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1CVu5m/build/debug/bin/postgresql.jar:/tmp/dist-test-task1CVu5m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 04:10:20,448 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 04:10:20,448 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1CVu5m/build/debug/bin/postgresql.jar:/tmp/dist-test-task1CVu5m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 04:10:20,678 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 04:10:20,679 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1CVu5m/build/debug/bin/postgresql.jar:/tmp/dist-test-task1CVu5m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 04:10:20,939 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1CVu5m/build/debug/bin/postgresql.jar:/tmp/dist-test-task1CVu5m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 04:10:21,167 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1CVu5m/build/debug/bin/postgresql.jar:/tmp/dist-test-task1CVu5m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-00ld.c.gcp-upstream.internal';"
2026-09-24 04:10:21,412 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 04:10:21,412 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1CVu5m/build/debug/bin/postgresql.jar:/tmp/dist-test-task1CVu5m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 04:10:21,646 [I] DB_PATCHES have already been applied
I20260924 04:10:21.652282 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ranger-admin:/tmp/dist-test-task1CVu5m/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task1CVu5m/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task1CVu5m/build/debug/bin/postgresql.jar:/tmp/dist-test-task1CVu5m/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 04:10:21.652388 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260924 04:10:21.655874 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:33403
Sep 24, 2026 4:10:21 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 4:10:21 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 4:10:22 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 4:10:22 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 4:10:22 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 4:10:22 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 4:10:22 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 04:10:22.759541 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.503s user 0.000s sys 0.010s
Sep 24, 2026 4:10:42 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 4:10:42 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 24, 2026 4:10:42 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 4:10:42 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 4:10:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 4:10:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 4:10:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 4:10:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 4:10:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 4:10:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 4:10:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 4:10:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 4:10:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 4:10:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 4:10:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 4:10:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 4:10:42 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 4:10:43 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 04:10:44.143466 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 04:10:44 dist-test-slave-00ld krb5kdc[12162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223044, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 04:10:44.229897 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 04:10:44.239466 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 04:10:45.138708 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 04:10:45.138819 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ranger-kms
I20260924 04:10:45.138842 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:57231
2026-09-24 04:10:45,307 [I] DB FLAVOR :POSTGRES
2026-09-24 04:10:45,307 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 04:10:45,308 [I] Checking connection
2026-09-24 04:10:45,308 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1CVu5m/build/debug/bin/postgresql.jar:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57231/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 04:10:45,572 [I] connection success
2026-09-24 04:10:45,572 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 04:10:45,573 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 04:10:45,573 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1CVu5m/build/debug/bin/postgresql.jar:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57231/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 04:10:45,838 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 04:10:45,839 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 04:10:45,839 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 04:10:45,839 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1CVu5m/build/debug/bin/postgresql.jar:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:57231/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 04:10:46,170 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 04:10:46.633569 392 mini_ranger_kms.cc:310] Created kms service
I20260924 04:10:46.827059 392 mini_ranger_kms.cc:326] Created kudu user
I20260924 04:10:46.870604 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 04:10:46 dist-test-slave-00ld krb5kdc[12162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223044, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 04:10:47.249127 392 mini_ranger_kms.cc:384] Added ranger policy
I20260924 04:10:47.249281 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ranger-kms:/tmp/dist-test-task1CVu5m/build/debug/bin/postgresql.jar:/tmp/dist-test-task1CVu5m/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task1CVu5m/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 04:10:47.249305 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260924 04:10:47.253188 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12555
I20260924 04:10:47.253283 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:35005
Sep 24, 2026 4:10:48 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 4:10:48 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 4:10:48 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 4:10:48 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 4:10:48 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 4:10:48 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260924 04:10:49.500360 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.270s user 0.005s sys 0.148s
I20260924 04:10:49.500487 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 04:10:49.500558 392 mini_ranger_kms.cc:401] 127.0.98.20:35005/kms/v1/keys
Sep 24 04:10:49 dist-test-slave-00ld krb5kdc[12162](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790223049, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 24 04:10:50 dist-test-slave-00ld krb5kdc[12162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223049, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 24 04:10:50 dist-test-slave-00ld krb5kdc[12162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223049, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 24 04:10:52 dist-test-slave-00ld krb5kdc[12162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223052, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:10:52.797139 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39255
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39255
--ranger_config_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:35005/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:10:52.907976 12606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:10:52.908241 12606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:10:52.908280 12606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:10:52.912245 12606 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:10:52.912308 12606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:10:52.912357 12606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:10:52.912438 12606 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:10:52.912489 12606 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:10:52.917936 12606 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:35005/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39255
--ranger_config_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39255
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12606
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:10:52.918955 12606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:10:52.919945 12606 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:10:52.928953 12612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:10:52.928972 12611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:10:52.929127 12606 server_base.cc:1034] running on GCE node
W20260924 04:10:52.929307 12614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:10:52.929855 12606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:10:52.930970 12606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:10:52.932156 12606 hybrid_clock.cc:648] HybridClock initialized: now 1790223052932141 us; error 31 us; skew 500 ppm
Sep 24 04:10:52 dist-test-slave-00ld krb5kdc[12162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223052, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:10:52.935022 12606 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:10:52.935989 12606 webserver.cc:460] Webserver started at http://127.0.98.62:36473/ using document root <none> and password file <none>
I20260924 04:10:52.936518 12606 fs_manager.cc:359] Metadata directory not provided
I20260924 04:10:52.936596 12606 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:10:52.936852 12606 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 04:10:52 dist-test-slave-00ld krb5kdc[12162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223052, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 04:10:52.995586 12606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/master-0/data/instance:
uuid: "ce511fd242bb4e408d08181ef52e1b32"
format_stamp: "Formatted at 2026-09-24 04:10:52 on dist-test-slave-00ld"
server_key: "e1e6646d82430288c761a8e620852ab8"
server_key_iv: "26c854c7aaf65a770d03f3b98caa5673"
server_key_version: "kuduclusterkey@0"
I20260924 04:10:52.996217 12606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/master-0/wal/instance:
uuid: "ce511fd242bb4e408d08181ef52e1b32"
format_stamp: "Formatted at 2026-09-24 04:10:52 on dist-test-slave-00ld"
server_key: "e1e6646d82430288c761a8e620852ab8"
server_key_iv: "26c854c7aaf65a770d03f3b98caa5673"
server_key_version: "kuduclusterkey@0"
I20260924 04:10:53.000218 12606 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 04:10:53.018198 12621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:10:53.019945 12606 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 04:10:53.020092 12606 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/master-0/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/master-0/wal
uuid: "ce511fd242bb4e408d08181ef52e1b32"
format_stamp: "Formatted at 2026-09-24 04:10:52 on dist-test-slave-00ld"
server_key: "e1e6646d82430288c761a8e620852ab8"
server_key_iv: "26c854c7aaf65a770d03f3b98caa5673"
server_key_version: "kuduclusterkey@0"
I20260924 04:10:53.020223 12606 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:10:53.046226 12606 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:10:53.049840 12606 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:10:53.050034 12606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:10:53.058450 12673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39255 every 8 connection(s)
I20260924 04:10:53.058473 12606 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39255
I20260924 04:10:53.059818 12606 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:10:53.062830 12674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:10:53.065279 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 12606
I20260924 04:10:53.065429 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/master-0/wal/instance
I20260924 04:10:53.065743 392 external_mini_cluster.cc:1428] Setting key cbcc4e47a86928a2ed4b82cc0aaf0092
I20260924 04:10:53.069324 12674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce511fd242bb4e408d08181ef52e1b32: Bootstrap starting.
I20260924 04:10:53.071877 12674 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ce511fd242bb4e408d08181ef52e1b32: Neither blocks nor log segments found. Creating new log.
I20260924 04:10:53.072849 12674 log.cc:824] T 00000000000000000000000000000000 P ce511fd242bb4e408d08181ef52e1b32: Log is configured to *not* fsync() on all Append() calls
I20260924 04:10:53.075145 12674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce511fd242bb4e408d08181ef52e1b32: No bootstrap required, opened a new log
Sep 24 04:10:53 dist-test-slave-00ld krb5kdc[12162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223052, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:10:53.079638 12674 raft_consensus.cc:348] T 00000000000000000000000000000000 P ce511fd242bb4e408d08181ef52e1b32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce511fd242bb4e408d08181ef52e1b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39255 } }
I20260924 04:10:53.080050 12674 raft_consensus.cc:374] T 00000000000000000000000000000000 P ce511fd242bb4e408d08181ef52e1b32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:10:53.080133 12674 raft_consensus.cc:729] T 00000000000000000000000000000000 P ce511fd242bb4e408d08181ef52e1b32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce511fd242bb4e408d08181ef52e1b32, State: Initialized, Role: FOLLOWER
I20260924 04:10:53.080653 12674 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce511fd242bb4e408d08181ef52e1b32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce511fd242bb4e408d08181ef52e1b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39255 } }
I20260924 04:10:53.080857 12674 raft_consensus.cc:388] T 00000000000000000000000000000000 P ce511fd242bb4e408d08181ef52e1b32 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:10:53.080960 12674 raft_consensus.cc:482] T 00000000000000000000000000000000 P ce511fd242bb4e408d08181ef52e1b32 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:10:53.081086 12674 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ce511fd242bb4e408d08181ef52e1b32 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:10:53.082209 12674 raft_consensus.cc:504] T 00000000000000000000000000000000 P ce511fd242bb4e408d08181ef52e1b32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce511fd242bb4e408d08181ef52e1b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39255 } }
I20260924 04:10:53.082569 12674 leader_election.cc:302] T 00000000000000000000000000000000 P ce511fd242bb4e408d08181ef52e1b32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ce511fd242bb4e408d08181ef52e1b32; no voters:
I20260924 04:10:53.082847 12677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:10:53.067364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47620 (local address 127.0.98.62:39255)
0924 04:10:53.067857 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0924 04:10:53.067877 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 04:10:53.067921 (+ 44us) server_negotiation.cc:407] Connection header received
0924 04:10:53.068832 (+ 911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:10:53.068859 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:10:53.069338 (+ 479us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:10:53.069726 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:10:53.070615 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:10:53.072079 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:10:53.073319 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:10:53.073895 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:10:53.074052 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:10:53.077293 (+ 3241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:10:53.077300 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:10:53.077329 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:10:53.077360 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:10:53.079467 (+ 2107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:10:53.079985 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:10:53.079991 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:10:53.079997 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:10:53.080134 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:10:53.080487 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:10:53.080490 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:10:53.080492 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:10:53.081070 (+ 578us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:10:53.081288 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0924 04:10:53.081563 (+ 275us) server_negotiation.cc:299] Negotiation successful
0924 04:10:53.081870 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":138,"threads_started":1}
I20260924 04:10:53.082962 12674 leader_election.cc:288] T 00000000000000000000000000000000 P ce511fd242bb4e408d08181ef52e1b32 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:10:53.083179 12679 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ce511fd242bb4e408d08181ef52e1b32 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:10:53.083359 12679 raft_consensus.cc:686] T 00000000000000000000000000000000 P ce511fd242bb4e408d08181ef52e1b32 [term 1 LEADER]: Becoming Leader. State: Replica: ce511fd242bb4e408d08181ef52e1b32, State: Running, Role: LEADER
I20260924 04:10:53.083729 12679 consensus_queue.cc:237] T 00000000000000000000000000000000 P ce511fd242bb4e408d08181ef52e1b32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce511fd242bb4e408d08181ef52e1b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39255 } }
I20260924 04:10:53.084301 12674 sys_catalog.cc:564] T 00000000000000000000000000000000 P ce511fd242bb4e408d08181ef52e1b32 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:10:53.084972 12680 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce511fd242bb4e408d08181ef52e1b32 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce511fd242bb4e408d08181ef52e1b32. Latest consensus state: current_term: 1 leader_uuid: "ce511fd242bb4e408d08181ef52e1b32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce511fd242bb4e408d08181ef52e1b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39255 } } }
I20260924 04:10:53.085090 12680 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce511fd242bb4e408d08181ef52e1b32 [sys.catalog]: This master's current role is: LEADER
I20260924 04:10:53.085477 12681 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce511fd242bb4e408d08181ef52e1b32 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ce511fd242bb4e408d08181ef52e1b32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce511fd242bb4e408d08181ef52e1b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39255 } } }
I20260924 04:10:53.085573 12681 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce511fd242bb4e408d08181ef52e1b32 [sys.catalog]: This master's current role is: LEADER
I20260924 04:10:53.085879 12685 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:10:53.088927 12685 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:10:53.089360 12674 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 04:10:53.097857 12685 catalog_manager.cc:1347] Generated new cluster ID: 05e6263987ad43b69f40eb9c4b7a302b
I20260924 04:10:53.097981 12685 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:10:53.113175 12685 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:10:53.114473 12685 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:10:53.122056 12685 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ce511fd242bb4e408d08181ef52e1b32: Generated new TSK 0
I20260924 04:10:53.122846 12685 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:10:53.780861 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:35005/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39255
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:10:53.926872 12719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:10:53.927238 12719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:10:53.927357 12719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:10:53.931648 12719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:10:53.931736 12719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:10:53.931830 12719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:10:53.937901 12719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:35005/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39255
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:10:53.939128 12719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:10:53.940208 12719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:10:53.951892 12724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:10:53.952461 12725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:10:53.952816 12727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:10:53.953543 12719 server_base.cc:1034] running on GCE node
I20260924 04:10:53.954149 12719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:10:53.954995 12719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:10:53.956220 12719 hybrid_clock.cc:648] HybridClock initialized: now 1790223053956201 us; error 44 us; skew 500 ppm
Sep 24 04:10:53 dist-test-slave-00ld krb5kdc[12162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223053, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:10:53.960032 12719 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:10:53.961149 12719 webserver.cc:460] Webserver started at http://127.0.98.1:41455/ using document root <none> and password file <none>
I20260924 04:10:53.961792 12719 fs_manager.cc:359] Metadata directory not provided
I20260924 04:10:53.961889 12719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:10:53.962141 12719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 04:10:53 dist-test-slave-00ld krb5kdc[12162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223053, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 04:10:53.979625 12719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ts-0/data/instance:
uuid: "21d1bee4a3ce411f8e564abfa332dcda"
format_stamp: "Formatted at 2026-09-24 04:10:53 on dist-test-slave-00ld"
server_key: "ab046266176e754e8fda501bd4f29f30"
server_key_iv: "29f71eefb52d3236a13cd57a27fadfcd"
server_key_version: "kuduclusterkey@0"
I20260924 04:10:53.980510 12719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ts-0/wal/instance:
uuid: "21d1bee4a3ce411f8e564abfa332dcda"
format_stamp: "Formatted at 2026-09-24 04:10:53 on dist-test-slave-00ld"
server_key: "ab046266176e754e8fda501bd4f29f30"
server_key_iv: "29f71eefb52d3236a13cd57a27fadfcd"
server_key_version: "kuduclusterkey@0"
I20260924 04:10:53.985925 12719 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 04:10:54.009222 12736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:10:54.011166 12719 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.002s sys 0.000s
I20260924 04:10:54.011343 12719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ts-0/wal
uuid: "21d1bee4a3ce411f8e564abfa332dcda"
format_stamp: "Formatted at 2026-09-24 04:10:53 on dist-test-slave-00ld"
server_key: "ab046266176e754e8fda501bd4f29f30"
server_key_iv: "29f71eefb52d3236a13cd57a27fadfcd"
server_key_version: "kuduclusterkey@0"
I20260924 04:10:54.011495 12719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:10:54.035208 12719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:10:54.039088 12719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:10:54.039366 12719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:10:54.040125 12719 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:10:54.041306 12719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:10:54.041416 12719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:10:54.041543 12719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:10:54.041625 12719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:10:54.052856 12719 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39949
I20260924 04:10:54.052955 12851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39949 every 8 connection(s)
I20260924 04:10:54.054131 12719 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:10:54.059780 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 12719
I20260924 04:10:54.059912 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:10:54.060254 392 external_mini_cluster.cc:1428] Setting key 812e484c3d445f64a5f07a31fed8b51a
Sep 24 04:10:54 dist-test-slave-00ld krb5kdc[12162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223053, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:10:54.074481 12854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:10:54.056551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38921 (local address 127.0.98.62:39255)
0924 04:10:54.057908 (+ 1357us) server_negotiation.cc:206] Beginning negotiation
0924 04:10:54.057917 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 04:10:54.059750 (+ 1833us) server_negotiation.cc:407] Connection header received
0924 04:10:54.059808 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:10:54.059814 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:10:54.059877 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:10:54.059977 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:10:54.062511 (+ 2534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:10:54.063394 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:10:54.065138 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:10:54.065485 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:10:54.065572 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:10:54.069924 (+ 4352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:10:54.069929 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:10:54.069932 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:10:54.069959 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:10:54.071879 (+ 1920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:10:54.072526 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:10:54.072530 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:10:54.072532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:10:54.072621 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:10:54.073132 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:10:54.073135 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:10:54.073137 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:10:54.073365 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:10:54.073450 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 04:10:54.074150 (+ 700us) server_negotiation.cc:299] Negotiation successful
0924 04:10:54.074281 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1265,"thread_start_us":128,"threads_started":1}
I20260924 04:10:54.076464 12855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:10:54.056791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39255 (local address 127.0.98.1:38921)
0924 04:10:54.058282 (+ 1491us) negotiation.cc:107] Waiting for socket to connect
0924 04:10:54.058342 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0924 04:10:54.059411 (+ 1069us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:10:54.060317 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:10:54.060328 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:10:54.060729 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:10:54.062334 (+ 1605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:10:54.062358 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:10:54.063582 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:10:54.063588 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:10:54.064997 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:10:54.065010 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:10:54.065794 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:10:54.065798 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:10:54.065998 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:10:54.067761 (+ 1763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:10:54.067785 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:10:54.069708 (+ 1923us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:10:54.072022 (+ 2314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:10:54.072028 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:10:54.072047 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:10:54.072395 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:10:54.072812 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:10:54.072816 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:10:54.072819 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:10:54.073014 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:10:54.073528 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:10:54.073535 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:10:54.073868 (+ 333us) client_negotiation.cc:769] Sending connection context
0924 04:10:54.074983 (+ 1115us) client_negotiation.cc:240] Negotiation successful
0924 04:10:54.075351 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1166,"thread_start_us":128,"threads_started":1}
I20260924 04:10:54.078208 12852 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39255
I20260924 04:10:54.078449 12852 heartbeater.cc:464] Registering TS with master...
I20260924 04:10:54.079015 12852 heartbeater.cc:511] Master 127.0.98.62:39255 requested a full tablet report, sending...
I20260924 04:10:54.080793 12635 ts_manager.cc:194] Registered new tserver with Master: 21d1bee4a3ce411f8e564abfa332dcda (127.0.98.1:39949)
I20260924 04:10:54.082284 12635 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38921
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:10:54.112521 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:35005/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39255
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 04:10:54 dist-test-slave-00ld krb5kdc[12162](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790223054, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260924 04:10:54.274906 12858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:10:54.275338 12858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:10:54.275416 12858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:10:54.281745 12858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:10:54.281828 12858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:10:54.281920 12858 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:10:54.290078 12858 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:35005/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39255
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12858
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:10:54.291725 12858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:10:54.293205 12858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:10:54.307097 12865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:10:54.308074 12867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:10:54.308238 12864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:10:54.308802 12858 server_base.cc:1034] running on GCE node
I20260924 04:10:54.309319 12858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:10:54.310107 12858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:10:54.311298 12858 hybrid_clock.cc:648] HybridClock initialized: now 1790223054311273 us; error 39 us; skew 500 ppm
Sep 24 04:10:54 dist-test-slave-00ld krb5kdc[12162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223054, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:10:54.315225 12858 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:10:54.316538 12858 webserver.cc:460] Webserver started at http://127.0.98.2:40811/ using document root <none> and password file <none>
I20260924 04:10:54.317339 12858 fs_manager.cc:359] Metadata directory not provided
I20260924 04:10:54.317492 12858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:10:54.317770 12858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 04:10:54 dist-test-slave-00ld krb5kdc[12162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223054, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 04:10:54.340747 12858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ts-1/data/instance:
uuid: "36578d05718744b28d577898616a97e8"
format_stamp: "Formatted at 2026-09-24 04:10:54 on dist-test-slave-00ld"
server_key: "007812a5be48ff30bff26419ac405b74"
server_key_iv: "31966c95714a92ec730106363eecddcd"
server_key_version: "kuduclusterkey@0"
I20260924 04:10:54.341765 12858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ts-1/wal/instance:
uuid: "36578d05718744b28d577898616a97e8"
format_stamp: "Formatted at 2026-09-24 04:10:54 on dist-test-slave-00ld"
server_key: "007812a5be48ff30bff26419ac405b74"
server_key_iv: "31966c95714a92ec730106363eecddcd"
server_key_version: "kuduclusterkey@0"
I20260924 04:10:54.347484 12858 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 04:10:54.371011 12874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:10:54.373133 12858 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.004s sys 0.000s
I20260924 04:10:54.373390 12858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ts-1/wal
uuid: "36578d05718744b28d577898616a97e8"
format_stamp: "Formatted at 2026-09-24 04:10:54 on dist-test-slave-00ld"
server_key: "007812a5be48ff30bff26419ac405b74"
server_key_iv: "31966c95714a92ec730106363eecddcd"
server_key_version: "kuduclusterkey@0"
I20260924 04:10:54.373617 12858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:10:54.456807 12858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:10:54.460951 12858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:10:54.461223 12858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:10:54.462075 12858 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:10:54.463265 12858 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:10:54.463344 12858 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:10:54.463404 12858 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:10:54.463447 12858 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:10:54.475136 12858 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33699
I20260924 04:10:54.476394 12858 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:10:54.478549 12987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33699 every 8 connection(s)
I20260924 04:10:54.483541 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 12858
I20260924 04:10:54.483669 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:10:54.484004 392 external_mini_cluster.cc:1428] Setting key 2a52388f9462d51a95d84e33866a715e
Sep 24 04:10:54 dist-test-slave-00ld krb5kdc[12162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223054, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 04:10:54.505836 12854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:10:54.481738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40827 (local address 127.0.98.62:39255)
0924 04:10:54.481861 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 04:10:54.481873 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 04:10:54.489261 (+ 7388us) server_negotiation.cc:407] Connection header received
0924 04:10:54.490689 (+ 1428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:10:54.490693 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:10:54.490750 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:10:54.490853 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:10:54.492463 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:10:54.493371 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:10:54.495065 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:10:54.495471 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:10:54.495601 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:10:54.501162 (+ 5561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:10:54.501166 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:10:54.501169 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:10:54.501200 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:10:54.502995 (+ 1795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:10:54.503662 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:10:54.503666 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:10:54.503668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:10:54.503761 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:10:54.504367 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:10:54.504370 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:10:54.504372 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:10:54.504610 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:10:54.504750 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 04:10:54.505414 (+ 664us) server_negotiation.cc:299] Negotiation successful
0924 04:10:54.505555 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260924 04:10:54.508262 12990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:10:54.481233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39255 (local address 127.0.98.2:40827)
0924 04:10:54.489055 (+ 7822us) negotiation.cc:107] Waiting for socket to connect
0924 04:10:54.489137 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0924 04:10:54.490397 (+ 1260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:10:54.491087 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:10:54.491099 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:10:54.491495 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:10:54.492291 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:10:54.492312 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:10:54.493524 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:10:54.493528 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:10:54.494919 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:10:54.494931 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:10:54.496761 (+ 1830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:10:54.496768 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:10:54.496921 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:10:54.499115 (+ 2194us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:10:54.499133 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:10:54.500915 (+ 1782us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:10:54.503151 (+ 2236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:10:54.503160 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:10:54.503181 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:10:54.503542 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:10:54.503893 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:10:54.503896 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:10:54.503898 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:10:54.504099 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:10:54.504760 (+ 661us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:10:54.504768 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:10:54.505114 (+ 346us) client_negotiation.cc:769] Sending connection context
0924 04:10:54.505347 (+ 233us) client_negotiation.cc:240] Negotiation successful
0924 04:10:54.505795 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7517,"thread_start_us":358,"threads_started":1}
I20260924 04:10:54.509750 12988 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39255
I20260924 04:10:54.510030 12988 heartbeater.cc:464] Registering TS with master...
I20260924 04:10:54.511937 12988 heartbeater.cc:511] Master 127.0.98.62:39255 requested a full tablet report, sending...
I20260924 04:10:54.513171 12635 ts_manager.cc:194] Registered new tserver with Master: 36578d05718744b28d577898616a97e8 (127.0.98.2:33699)
I20260924 04:10:54.513919 12635 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:40827
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:10:54.529603 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:35005/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39255
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 04:10:54 dist-test-slave-00ld krb5kdc[12162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223054, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260924 04:10:54.697794 12993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:10:54.698202 12993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:10:54.698287 12993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:10:54.704207 12993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:10:54.704303 12993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:10:54.704388 12993 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:10:54.711938 12993 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:35005/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39255
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12993
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:10:54.713440 12993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:10:54.714847 12993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:10:54.727406 13001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:10:54.727456 12998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:10:54.727789 12999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:10:54.727882 12993 server_base.cc:1034] running on GCE node
I20260924 04:10:54.728439 12993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:10:54.729225 12993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:10:54.730402 12993 hybrid_clock.cc:648] HybridClock initialized: now 1790223054730372 us; error 49 us; skew 500 ppm
Sep 24 04:10:54 dist-test-slave-00ld krb5kdc[12162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223054, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:10:54.734141 12993 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:10:54.735481 12993 webserver.cc:460] Webserver started at http://127.0.98.3:42017/ using document root <none> and password file <none>
I20260924 04:10:54.736203 12993 fs_manager.cc:359] Metadata directory not provided
I20260924 04:10:54.736299 12993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:10:54.736547 12993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 04:10:54 dist-test-slave-00ld krb5kdc[12162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223054, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 04:10:54.758841 12993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ts-2/data/instance:
uuid: "c7f4b9eb1354489f92a32f227eb109de"
format_stamp: "Formatted at 2026-09-24 04:10:54 on dist-test-slave-00ld"
server_key: "54f3635f2de2d2727db8612b4b3b1a4d"
server_key_iv: "bb3bb5add371cfacc865a0ba1d1d4e19"
server_key_version: "kuduclusterkey@0"
I20260924 04:10:54.759523 12993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c7f4b9eb1354489f92a32f227eb109de"
format_stamp: "Formatted at 2026-09-24 04:10:54 on dist-test-slave-00ld"
server_key: "54f3635f2de2d2727db8612b4b3b1a4d"
server_key_iv: "bb3bb5add371cfacc865a0ba1d1d4e19"
server_key_version: "kuduclusterkey@0"
I20260924 04:10:54.763789 12993 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260924 04:10:54.784729 13008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:10:54.786600 12993 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.002s sys 0.001s
I20260924 04:10:54.786744 12993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ts-2/wal
uuid: "c7f4b9eb1354489f92a32f227eb109de"
format_stamp: "Formatted at 2026-09-24 04:10:54 on dist-test-slave-00ld"
server_key: "54f3635f2de2d2727db8612b4b3b1a4d"
server_key_iv: "bb3bb5add371cfacc865a0ba1d1d4e19"
server_key_version: "kuduclusterkey@0"
I20260924 04:10:54.786892 12993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:10:54.809602 12993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:10:54.812791 12993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:10:54.813014 12993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:10:54.813642 12993 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:10:54.814582 12993 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:10:54.814654 12993 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:10:54.814754 12993 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:10:54.814800 12993 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:10:54.826068 12993 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41761
I20260924 04:10:54.827018 12993 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:10:54.832105 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 12993
I20260924 04:10:54.832252 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:10:54.832597 392 external_mini_cluster.cc:1428] Setting key 7ed9497507c8f85857924b0161113067
Sep 24 04:10:54 dist-test-slave-00ld krb5kdc[12162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223054, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:10:54.841156 13121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41761 every 8 connection(s)
I20260924 04:10:54.842820 12854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:10:54.828974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48765 (local address 127.0.98.62:39255)
0924 04:10:54.829104 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 04:10:54.829109 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:10:54.830064 (+ 955us) server_negotiation.cc:407] Connection header received
0924 04:10:54.831049 (+ 985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:10:54.831053 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:10:54.831108 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:10:54.831215 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:10:54.832557 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:10:54.833331 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:10:54.834744 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:10:54.835034 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:10:54.835096 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:10:54.838793 (+ 3697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:10:54.838797 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:10:54.838800 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:10:54.838838 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:10:54.840422 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:10:54.841102 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:10:54.841105 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:10:54.841107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:10:54.841198 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:10:54.841621 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:10:54.841624 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:10:54.841626 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:10:54.841862 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:10:54.841937 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 04:10:54.842534 (+ 597us) server_negotiation.cc:299] Negotiation successful
0924 04:10:54.842655 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 04:10:54.843722 13124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:10:54.829447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39255 (local address 127.0.98.3:48765)
0924 04:10:54.829910 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0924 04:10:54.829963 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 04:10:54.830847 (+ 884us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:10:54.831387 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:10:54.831394 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:10:54.831658 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:10:54.832376 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:10:54.832398 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:10:54.833501 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:10:54.833504 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:10:54.834617 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:10:54.834629 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:10:54.835251 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:10:54.835254 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:10:54.835418 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:10:54.836026 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:10:54.836042 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:10:54.838587 (+ 2545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:10:54.840556 (+ 1969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:10:54.840563 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:10:54.840581 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:10:54.840971 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:10:54.841319 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:10:54.841322 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:10:54.841324 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:10:54.841507 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:10:54.842023 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:10:54.842035 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:10:54.842288 (+ 253us) client_negotiation.cc:769] Sending connection context
0924 04:10:54.842481 (+ 193us) client_negotiation.cc:240] Negotiation successful
0924 04:10:54.842798 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":125,"threads_started":1}
I20260924 04:10:54.845237 13122 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39255
I20260924 04:10:54.845485 13122 heartbeater.cc:464] Registering TS with master...
I20260924 04:10:54.846000 13122 heartbeater.cc:511] Master 127.0.98.62:39255 requested a full tablet report, sending...
I20260924 04:10:54.847107 12635 ts_manager.cc:194] Registered new tserver with Master: c7f4b9eb1354489f92a32f227eb109de (127.0.98.3:41761)
I20260924 04:10:54.847746 12635 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48765
I20260924 04:10:54.848070 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:10:54.849014 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 12719
I20260924 04:10:54.855340 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 12858
I20260924 04:10:54.861140 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 12993
I20260924 04:10:54.867651 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 12606
I20260924 04:10:54.874891 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39255
--webserver_interface=127.0.98.62
--webserver_port=36473
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39255
--ranger_config_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:35005/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:35005/kms with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:10:54.988484 13125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:10:54.988802 13125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:10:54.988885 13125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:10:54.993152 13125 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:10:54.993212 13125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:10:54.993232 13125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:10:54.993249 13125 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:10:54.993304 13125 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:10:54.998996 13125 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:35005/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39255
--ranger_config_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39255
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=36473
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13125
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:10:55.000097 13125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:10:55.001248 13125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:10:55.010005 13133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:10:55.010040 13130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:10:55.010008 13131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:10:55.010335 13125 server_base.cc:1034] running on GCE node
I20260924 04:10:55.010939 13125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:10:55.012029 13125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:10:55.013212 13125 hybrid_clock.cc:648] HybridClock initialized: now 1790223055013177 us; error 52 us; skew 500 ppm
Sep 24 04:10:55 dist-test-slave-00ld krb5kdc[12162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223055, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:10:55.015879 13125 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:10:55.016837 13125 webserver.cc:460] Webserver started at http://127.0.98.62:36473/ using document root <none> and password file <none>
I20260924 04:10:55.017370 13125 fs_manager.cc:359] Metadata directory not provided
I20260924 04:10:55.017494 13125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 04:10:55 dist-test-slave-00ld krb5kdc[12162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223055, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 04:10:55.078531 13125 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 04:10:55.081292 13141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:10:55.082962 13125 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:10:55.083108 13125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/master-0/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/master-0/wal
uuid: "ce511fd242bb4e408d08181ef52e1b32"
format_stamp: "Formatted at 2026-09-24 04:10:52 on dist-test-slave-00ld"
server_key: "e1e6646d82430288c761a8e620852ab8"
server_key_iv: "26c854c7aaf65a770d03f3b98caa5673"
server_key_version: "kuduclusterkey@0"
I20260924 04:10:55.083695 13125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:10:55.097437 13125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:10:55.100277 13125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:10:55.100507 13125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:10:55.107702 13125 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39255
I20260924 04:10:55.107707 13193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39255 every 8 connection(s)
I20260924 04:10:55.108779 13125 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:10:55.112659 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 13125
I20260924 04:10:55.113019 13194 sys_catalog.cc:263] Verifying existing consensus state
I20260924 04:10:55.113207 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:35005/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:39949
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:39255
--webserver_port=41455
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:35005/kms with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:10:55.114709 13194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce511fd242bb4e408d08181ef52e1b32: Bootstrap starting.
I20260924 04:10:55.130722 13194 log.cc:824] T 00000000000000000000000000000000 P ce511fd242bb4e408d08181ef52e1b32: Log is configured to *not* fsync() on all Append() calls
I20260924 04:10:55.137200 13194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce511fd242bb4e408d08181ef52e1b32: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 04:10:55.137593 13194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce511fd242bb4e408d08181ef52e1b32: Bootstrap complete.
I20260924 04:10:55.141697 13194 raft_consensus.cc:348] T 00000000000000000000000000000000 P ce511fd242bb4e408d08181ef52e1b32 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce511fd242bb4e408d08181ef52e1b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39255 } }
I20260924 04:10:55.142104 13194 raft_consensus.cc:729] T 00000000000000000000000000000000 P ce511fd242bb4e408d08181ef52e1b32 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce511fd242bb4e408d08181ef52e1b32, State: Initialized, Role: FOLLOWER
I20260924 04:10:55.142584 13194 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce511fd242bb4e408d08181ef52e1b32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce511fd242bb4e408d08181ef52e1b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39255 } }
I20260924 04:10:55.142724 13194 raft_consensus.cc:388] T 00000000000000000000000000000000 P ce511fd242bb4e408d08181ef52e1b32 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:10:55.142810 13194 raft_consensus.cc:482] T 00000000000000000000000000000000 P ce511fd242bb4e408d08181ef52e1b32 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:10:55.142932 13194 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ce511fd242bb4e408d08181ef52e1b32 [term 1 FOLLOWER]: Advancing to term 2
I20260924 04:10:55.144815 13194 raft_consensus.cc:504] T 00000000000000000000000000000000 P ce511fd242bb4e408d08181ef52e1b32 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce511fd242bb4e408d08181ef52e1b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39255 } }
I20260924 04:10:55.145140 13194 leader_election.cc:302] T 00000000000000000000000000000000 P ce511fd242bb4e408d08181ef52e1b32 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ce511fd242bb4e408d08181ef52e1b32; no voters:
I20260924 04:10:55.145448 13194 leader_election.cc:288] T 00000000000000000000000000000000 P ce511fd242bb4e408d08181ef52e1b32 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 04:10:55.145573 13198 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ce511fd242bb4e408d08181ef52e1b32 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 04:10:55.145807 13198 raft_consensus.cc:686] T 00000000000000000000000000000000 P ce511fd242bb4e408d08181ef52e1b32 [term 2 LEADER]: Becoming Leader. State: Replica: ce511fd242bb4e408d08181ef52e1b32, State: Running, Role: LEADER
I20260924 04:10:55.146200 13198 consensus_queue.cc:237] T 00000000000000000000000000000000 P ce511fd242bb4e408d08181ef52e1b32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce511fd242bb4e408d08181ef52e1b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39255 } }
I20260924 04:10:55.146648 13194 sys_catalog.cc:564] T 00000000000000000000000000000000 P ce511fd242bb4e408d08181ef52e1b32 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:10:55.147253 13198 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce511fd242bb4e408d08181ef52e1b32 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce511fd242bb4e408d08181ef52e1b32. Latest consensus state: current_term: 2 leader_uuid: "ce511fd242bb4e408d08181ef52e1b32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce511fd242bb4e408d08181ef52e1b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39255 } } }
I20260924 04:10:55.147362 13198 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce511fd242bb4e408d08181ef52e1b32 [sys.catalog]: This master's current role is: LEADER
I20260924 04:10:55.147707 13204 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:10:55.147792 13199 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce511fd242bb4e408d08181ef52e1b32 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ce511fd242bb4e408d08181ef52e1b32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce511fd242bb4e408d08181ef52e1b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39255 } } }
I20260924 04:10:55.147969 13199 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce511fd242bb4e408d08181ef52e1b32 [sys.catalog]: This master's current role is: LEADER
I20260924 04:10:55.150900 13204 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:10:55.151399 13194 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 04:10:55.151547 13204 catalog_manager.cc:1259] Loaded cluster ID: 05e6263987ad43b69f40eb9c4b7a302b
I20260924 04:10:55.151593 13204 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:10:55.156570 13204 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:10:55.158429 13204 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ce511fd242bb4e408d08181ef52e1b32: Loaded TSK: 0
I20260924 04:10:55.160143 13204 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 04:10:55.258392 13196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:10:55.258759 13196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:10:55.258819 13196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:10:55.265465 13196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:10:55.265599 13196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:10:55.265707 13196 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:10:55.273950 13196 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:35005/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:39949
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=41455
--tserver_master_addrs=127.0.98.62:39255
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13196
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:10:55.275583 13196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:10:55.277192 13196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:10:55.289769 13232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:10:55.289855 13231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:10:55.289769 13234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:10:55.290235 13196 server_base.cc:1034] running on GCE node
I20260924 04:10:55.290700 13196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:10:55.291513 13196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:10:55.292742 13196 hybrid_clock.cc:648] HybridClock initialized: now 1790223055292720 us; error 35 us; skew 500 ppm
Sep 24 04:10:55 dist-test-slave-00ld krb5kdc[12162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223055, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:10:55.296736 13196 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:10:55.298020 13196 webserver.cc:460] Webserver started at http://127.0.98.1:41455/ using document root <none> and password file <none>
I20260924 04:10:55.298780 13196 fs_manager.cc:359] Metadata directory not provided
I20260924 04:10:55.298949 13196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 04:10:55 dist-test-slave-00ld krb5kdc[12162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223055, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 04:10:55.324962 13196 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 04:10:55.327836 13242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:10:55.329725 13196 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 04:10:55.329885 13196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ts-0/wal
uuid: "21d1bee4a3ce411f8e564abfa332dcda"
format_stamp: "Formatted at 2026-09-24 04:10:53 on dist-test-slave-00ld"
server_key: "ab046266176e754e8fda501bd4f29f30"
server_key_iv: "29f71eefb52d3236a13cd57a27fadfcd"
server_key_version: "kuduclusterkey@0"
I20260924 04:10:55.330699 13196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:10:55.361821 13196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:10:55.365643 13196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:10:55.365859 13196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:10:55.366545 13196 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:10:55.367584 13196 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:10:55.367651 13196 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:10:55.367709 13196 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:10:55.367790 13196 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260924 04:10:55.379585 13196 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39949
I20260924 04:10:55.380959 13196 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:10:55.382855 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 13196
I20260924 04:10:55.382989 13355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39949 every 8 connection(s)
I20260924 04:10:55.383492 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:35005/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:33699
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:39255
--webserver_port=40811
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:35005/kms with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 04:10:55 dist-test-slave-00ld krb5kdc[12162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223055, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:10:55.408902 13359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:10:55.385945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39255 (local address 127.0.98.1:38187)
0924 04:10:55.387548 (+ 1603us) negotiation.cc:107] Waiting for socket to connect
0924 04:10:55.387612 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0924 04:10:55.389191 (+ 1579us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:10:55.392125 (+ 2934us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:10:55.392138 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:10:55.392535 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:10:55.393378 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:10:55.393404 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:10:55.395944 (+ 2540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:10:55.395948 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:10:55.397347 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:10:55.397358 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:10:55.398420 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:10:55.398424 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:10:55.398608 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:10:55.399305 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:10:55.399325 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:10:55.402093 (+ 2768us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:10:55.405130 (+ 3037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:10:55.405139 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:10:55.405156 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:10:55.405511 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:10:55.405919 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:10:55.405922 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:10:55.405924 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:10:55.406093 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:10:55.407018 (+ 925us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:10:55.407025 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:10:55.407325 (+ 300us) client_negotiation.cc:769] Sending connection context
0924 04:10:55.407499 (+ 174us) client_negotiation.cc:240] Negotiation successful
0924 04:10:55.407831 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1353,"thread_start_us":149,"threads_started":1}
I20260924 04:10:55.410974 13360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:10:55.389878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38187 (local address 127.0.98.62:39255)
0924 04:10:55.390576 (+ 698us) server_negotiation.cc:206] Beginning negotiation
0924 04:10:55.390605 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0924 04:10:55.390686 (+ 81us) server_negotiation.cc:407] Connection header received
0924 04:10:55.391070 (+ 384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:10:55.391089 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:10:55.391569 (+ 480us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:10:55.393667 (+ 2098us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:10:55.394045 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:10:55.395771 (+ 1726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:10:55.397509 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:10:55.398241 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:10:55.399005 (+ 764us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:10:55.402314 (+ 3309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:10:55.402320 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:10:55.402337 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:10:55.402370 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:10:55.404976 (+ 2606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:10:55.405652 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:10:55.405658 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:10:55.405664 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:10:55.405802 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:10:55.406199 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:10:55.406202 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:10:55.406204 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:10:55.406889 (+ 685us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:10:55.409613 (+ 2724us) server_negotiation.cc:1035] Waiting for connection context
0924 04:10:55.409767 (+ 154us) server_negotiation.cc:299] Negotiation successful
0924 04:10:55.410079 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":135,"threads_started":1}
I20260924 04:10:55.413757 13356 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39255
I20260924 04:10:55.414017 13356 heartbeater.cc:464] Registering TS with master...
I20260924 04:10:55.414565 13356 heartbeater.cc:511] Master 127.0.98.62:39255 requested a full tablet report, sending...
W20260924 04:10:55.416923 13356 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39255 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 04:10:55.417039 13356 heartbeater.cc:464] Registering TS with master...
I20260924 04:10:55.417202 13356 heartbeater.cc:511] Master 127.0.98.62:39255 requested a full tablet report, sending...
I20260924 04:10:55.418228 13356 heartbeater.cc:464] Registering TS with master...
I20260924 04:10:55.418390 13356 heartbeater.cc:511] Master 127.0.98.62:39255 requested a full tablet report, sending...
W20260924 04:10:55.419186 13356 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 04:10:55.525240 13358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:10:55.525576 13358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:10:55.525689 13358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:10:55.530189 13358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:10:55.530267 13358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:10:55.530403 13358 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:10:55.536191 13358 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:35005/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:33699
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=40811
--tserver_master_addrs=127.0.98.62:39255
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13358
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:10:55.537585 13358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:10:55.538617 13358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:10:55.549489 13365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:10:55.549476 13366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:10:55.549762 13368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:10:55.549882 13358 server_base.cc:1034] running on GCE node
I20260924 04:10:55.550297 13358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:10:55.550951 13358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:10:55.552141 13358 hybrid_clock.cc:648] HybridClock initialized: now 1790223055552100 us; error 55 us; skew 500 ppm
Sep 24 04:10:55 dist-test-slave-00ld krb5kdc[12162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223055, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:10:55.555567 13358 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:10:55.556749 13358 webserver.cc:460] Webserver started at http://127.0.98.2:40811/ using document root <none> and password file <none>
I20260924 04:10:55.557539 13358 fs_manager.cc:359] Metadata directory not provided
I20260924 04:10:55.557693 13358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 04:10:55 dist-test-slave-00ld krb5kdc[12162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223055, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 04:10:55.580312 13358 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.002s sys 0.000s
I20260924 04:10:55.589480 13376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:10:55.591250 13358 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.002s sys 0.000s
I20260924 04:10:55.591419 13358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ts-1/wal
uuid: "36578d05718744b28d577898616a97e8"
format_stamp: "Formatted at 2026-09-24 04:10:54 on dist-test-slave-00ld"
server_key: "007812a5be48ff30bff26419ac405b74"
server_key_iv: "31966c95714a92ec730106363eecddcd"
server_key_version: "kuduclusterkey@0"
I20260924 04:10:55.592187 13358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:10:55.610136 13358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:10:55.613592 13358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:10:55.613806 13358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:10:55.614527 13358 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:10:55.615633 13358 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:10:55.615701 13358 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:10:55.615759 13358 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:10:55.615793 13358 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:10:55.625293 13358 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33699
I20260924 04:10:55.625312 13489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33699 every 8 connection(s)
I20260924 04:10:55.626397 13358 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:10:55.629441 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 13358
I20260924 04:10:55.629971 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:35005/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:41761
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:39255
--webserver_port=42017
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:35005/kms with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 04:10:55 dist-test-slave-00ld krb5kdc[12162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223055, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:10:55.648648 13360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:10:55.628914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41683 (local address 127.0.98.62:39255)
0924 04:10:55.629052 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 04:10:55.629057 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:10:55.629442 (+ 385us) server_negotiation.cc:407] Connection header received
0924 04:10:55.630574 (+ 1132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:10:55.630577 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:10:55.630640 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:10:55.630716 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:10:55.632023 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:10:55.632942 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:10:55.634177 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:10:55.634633 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:10:55.634764 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:10:55.639847 (+ 5083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:10:55.639854 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:10:55.639857 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:10:55.639895 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:10:55.645846 (+ 5951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:10:55.646618 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:10:55.646622 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:10:55.646624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:10:55.646716 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:10:55.647175 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:10:55.647178 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:10:55.647180 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:10:55.647402 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:10:55.647470 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 04:10:55.648320 (+ 850us) server_negotiation.cc:299] Negotiation successful
0924 04:10:55.648463 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 04:10:55.650640 13492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:10:55.628579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39255 (local address 127.0.98.2:41683)
0924 04:10:55.629292 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0924 04:10:55.629342 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 04:10:55.630357 (+ 1015us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:10:55.630889 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:10:55.630895 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:10:55.631162 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:10:55.631827 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:10:55.631850 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:10:55.633083 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:10:55.633085 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:10:55.634046 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:10:55.634055 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:10:55.636834 (+ 2779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:10:55.636840 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:10:55.637097 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:10:55.637919 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:10:55.637946 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:10:55.639612 (+ 1666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:10:55.646102 (+ 6490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:10:55.646109 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:10:55.646128 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:10:55.646486 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:10:55.646884 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:10:55.646887 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:10:55.646889 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:10:55.647068 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:10:55.647562 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:10:55.647569 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:10:55.648011 (+ 442us) client_negotiation.cc:769] Sending connection context
0924 04:10:55.649069 (+ 1058us) client_negotiation.cc:240] Negotiation successful
0924 04:10:55.649570 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":126,"threads_started":1}
I20260924 04:10:55.652081 13490 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39255
I20260924 04:10:55.652318 13490 heartbeater.cc:464] Registering TS with master...
I20260924 04:10:55.652921 13490 heartbeater.cc:511] Master 127.0.98.62:39255 requested a full tablet report, sending...
W20260924 04:10:55.654600 13490 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39255 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 04:10:55.654726 13490 heartbeater.cc:464] Registering TS with master...
I20260924 04:10:55.654875 13490 heartbeater.cc:511] Master 127.0.98.62:39255 requested a full tablet report, sending...
I20260924 04:10:55.655680 13490 heartbeater.cc:464] Registering TS with master...
I20260924 04:10:55.655844 13490 heartbeater.cc:511] Master 127.0.98.62:39255 requested a full tablet report, sending...
W20260924 04:10:55.656553 13490 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 04:10:55.782390 13493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:10:55.782689 13493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:10:55.782783 13493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:10:55.787195 13493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:10:55.787319 13493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:10:55.787416 13493 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:10:55.793777 13493 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:35005/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:41761
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=42017
--tserver_master_addrs=127.0.98.62:39255
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13493
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:10:55.795318 13493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:10:55.796841 13493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:10:55.810123 13499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:10:55.810130 13502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:10:55.810729 13500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:10:55.811761 13493 server_base.cc:1034] running on GCE node
I20260924 04:10:55.812273 13493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:10:55.813140 13493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:10:55.814409 13493 hybrid_clock.cc:648] HybridClock initialized: now 1790223055814345 us; error 80 us; skew 500 ppm
Sep 24 04:10:55 dist-test-slave-00ld krb5kdc[12162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223055, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:10:55.817332 13493 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:10:55.818351 13493 webserver.cc:460] Webserver started at http://127.0.98.3:42017/ using document root <none> and password file <none>
I20260924 04:10:55.819063 13493 fs_manager.cc:359] Metadata directory not provided
I20260924 04:10:55.819175 13493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 04:10:55 dist-test-slave-00ld krb5kdc[12162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223055, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 04:10:55.841658 13493 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.002s sys 0.000s
I20260924 04:10:55.845178 13510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:10:55.846896 13493 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 04:10:55.847088 13493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ts-2/wal
uuid: "c7f4b9eb1354489f92a32f227eb109de"
format_stamp: "Formatted at 2026-09-24 04:10:54 on dist-test-slave-00ld"
server_key: "54f3635f2de2d2727db8612b4b3b1a4d"
server_key_iv: "bb3bb5add371cfacc865a0ba1d1d4e19"
server_key_version: "kuduclusterkey@0"
I20260924 04:10:55.847832 13493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:10:55.868566 13493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:10:55.872396 13493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:10:55.872618 13493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:10:55.873389 13493 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:10:55.874455 13493 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:10:55.874563 13493 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:10:55.874667 13493 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:10:55.874740 13493 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:10:55.886759 13493 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41761
I20260924 04:10:55.888401 13493 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:10:55.890134 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 13493
I20260924 04:10:55.896327 13623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41761 every 8 connection(s)
Sep 24 04:10:55 dist-test-slave-00ld krb5kdc[12162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223055, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:10:55.910600 13360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:10:55.891220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47628 (local address 127.0.98.62:39255)
0924 04:10:55.891372 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 04:10:55.891377 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:10:55.897515 (+ 6138us) server_negotiation.cc:407] Connection header received
0924 04:10:55.897655 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:10:55.897658 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:10:55.897713 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:10:55.897829 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:10:55.901808 (+ 3979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:10:55.902602 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:10:55.903485 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:10:55.903820 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:10:55.903908 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:10:55.906412 (+ 2504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:10:55.906415 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:10:55.906418 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:10:55.906450 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:10:55.908029 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:10:55.908491 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:10:55.908495 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:10:55.908497 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:10:55.908575 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:10:55.909070 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:10:55.909074 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:10:55.909075 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:10:55.909241 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:10:55.909444 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0924 04:10:55.910327 (+ 883us) server_negotiation.cc:299] Negotiation successful
0924 04:10:55.910440 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 04:10:55.914407 13628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:10:55.894162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43915 (local address 127.0.98.62:39255)
0924 04:10:55.903598 (+ 9436us) server_negotiation.cc:206] Beginning negotiation
0924 04:10:55.903602 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:10:55.903612 (+ 10us) server_negotiation.cc:407] Connection header received
0924 04:10:55.903641 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:10:55.903643 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:10:55.903686 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:10:55.903763 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:10:55.905182 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:10:55.905925 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:10:55.907611 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:10:55.907979 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:10:55.908096 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:10:55.910733 (+ 2637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:10:55.910736 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:10:55.910740 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:10:55.910762 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:10:55.912362 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:10:55.912937 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:10:55.912940 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:10:55.912942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:10:55.913025 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:10:55.913358 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:10:55.913361 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:10:55.913362 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:10:55.913531 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:10:55.913636 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 04:10:55.914131 (+ 495us) server_negotiation.cc:299] Negotiation successful
0924 04:10:55.914239 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9364,"thread_start_us":95,"threads_started":1}
I20260924 04:10:55.915262 13627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:10:55.893115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39255 (local address 127.0.98.3:43915)
0924 04:10:55.897976 (+ 4861us) negotiation.cc:107] Waiting for socket to connect
0924 04:10:55.898036 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0924 04:10:55.899104 (+ 1068us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:10:55.904074 (+ 4970us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:10:55.904082 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:10:55.904348 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:10:55.905012 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:10:55.905035 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:10:55.906095 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:10:55.906099 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:10:55.907482 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:10:55.907493 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:10:55.908152 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:10:55.908157 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:10:55.908364 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:10:55.908943 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:10:55.908974 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:10:55.910554 (+ 1580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:10:55.912499 (+ 1945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:10:55.912505 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:10:55.912526 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:10:55.912836 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:10:55.913115 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:10:55.913120 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:10:55.913122 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:10:55.913260 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:10:55.913634 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:10:55.913640 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:10:55.913894 (+ 254us) client_negotiation.cc:769] Sending connection context
0924 04:10:55.914078 (+ 184us) client_negotiation.cc:240] Negotiation successful
0924 04:10:55.914419 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4562,"thread_start_us":145,"threads_started":1}
I20260924 04:10:55.916366 13624 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39255
I20260924 04:10:55.916633 13624 heartbeater.cc:464] Registering TS with master...
I20260924 04:10:55.917224 13624 heartbeater.cc:511] Master 127.0.98.62:39255 requested a full tablet report, sending...
W20260924 04:10:55.918857 13624 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39255 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 04:10:55.918987 13624 heartbeater.cc:464] Registering TS with master...
I20260924 04:10:55.919165 13624 heartbeater.cc:511] Master 127.0.98.62:39255 requested a full tablet report, sending...
I20260924 04:10:55.919988 13624 heartbeater.cc:464] Registering TS with master...
I20260924 04:10:55.920167 13624 heartbeater.cc:511] Master 127.0.98.62:39255 requested a full tablet report, sending...
W20260924 04:10:55.920905 13624 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 24 04:10:56 dist-test-slave-00ld krb5kdc[12162](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790223056, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:10:56.419414 13356 heartbeater.cc:464] Registering TS with master...
I20260924 04:10:56.419720 13356 heartbeater.cc:511] Master 127.0.98.62:39255 requested a full tablet report, sending...
I20260924 04:10:56.420925 13158 ts_manager.cc:194] Registered new tserver with Master: 21d1bee4a3ce411f8e564abfa332dcda (127.0.98.1:39949)
I20260924 04:10:56.422566 13158 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38187
Sep 24 04:10:56 dist-test-slave-00ld krb5kdc[12162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223056, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 04:10:56.656790 13490 heartbeater.cc:464] Registering TS with master...
I20260924 04:10:56.657095 13490 heartbeater.cc:511] Master 127.0.98.62:39255 requested a full tablet report, sending...
I20260924 04:10:56.657871 13158 ts_manager.cc:194] Registered new tserver with Master: 36578d05718744b28d577898616a97e8 (127.0.98.2:33699)
I20260924 04:10:56.658475 13158 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41683
Sep 24 04:10:56 dist-test-slave-00ld krb5kdc[12162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223056, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 04:10:56.921180 13624 heartbeater.cc:464] Registering TS with master...
I20260924 04:10:56.921509 13624 heartbeater.cc:511] Master 127.0.98.62:39255 requested a full tablet report, sending...
I20260924 04:10:56.922191 13158 ts_manager.cc:194] Registered new tserver with Master: c7f4b9eb1354489f92a32f227eb109de (127.0.98.3:41761)
I20260924 04:10:56.922780 13158 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:43915
I20260924 04:10:56.930553 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:10:56.964272 13647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:10:56.932433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47630 (local address 127.0.98.62:39255)
0924 04:10:56.937385 (+ 4952us) server_negotiation.cc:206] Beginning negotiation
0924 04:10:56.937390 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:10:56.944533 (+ 7143us) server_negotiation.cc:407] Connection header received
0924 04:10:56.944594 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:10:56.944597 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:10:56.944657 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:10:56.944776 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:10:56.945785 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:10:56.946658 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:10:56.947860 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:10:56.948204 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:10:56.948306 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:10:56.951619 (+ 3313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:10:56.951623 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:10:56.951626 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:10:56.951656 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:10:56.961677 (+ 10021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:10:56.962399 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:10:56.962403 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:10:56.962405 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:10:56.962528 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:10:56.962822 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:10:56.962825 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:10:56.962827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:10:56.963061 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:10:56.963191 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0924 04:10:56.963941 (+ 750us) server_negotiation.cc:299] Negotiation successful
0924 04:10:56.964081 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":85,"threads_started":1}
I20260924 04:10:56.968979 13158 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47630:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:10:56.974172 13158 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:10:57.006286 13654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:10:56.995748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41761 (local address 127.0.0.1:37614)
0924 04:10:56.998755 (+ 3007us) negotiation.cc:107] Waiting for socket to connect
0924 04:10:56.998769 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:10:56.998856 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:10:56.999591 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:10:56.999598 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:10:56.999621 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:10:56.999749 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:10:56.999755 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:10:57.002082 (+ 2327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:10:57.002086 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:10:57.004107 (+ 2021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:10:57.004119 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:10:57.005686 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:10:57.005690 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:10:57.005827 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:10:57.005870 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 04:10:57.005991 (+ 121us) client_negotiation.cc:240] Negotiation successful
0924 04:10:57.006122 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2947,"thread_start_us":93,"threads_started":1}
I20260924 04:10:57.006950 13657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:10:56.996133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37614 (local address 127.0.98.3:41761)
0924 04:10:56.998618 (+ 2485us) server_negotiation.cc:206] Beginning negotiation
0924 04:10:56.998628 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 04:10:56.998950 (+ 322us) server_negotiation.cc:407] Connection header received
0924 04:10:56.999030 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:10:56.999035 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:10:56.999241 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:10:56.999385 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:10:57.000597 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:10:57.001898 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:10:57.004260 (+ 2362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:10:57.005309 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:10:57.005439 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:10:57.005612 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0924 04:10:57.006670 (+ 1058us) server_negotiation.cc:299] Negotiation successful
0924 04:10:57.006784 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2354,"thread_start_us":98,"threads_started":1}
I20260924 04:10:57.011355 13555 tablet_service.cc:1467] Processing CreateTablet for tablet 8fcd60b69247467fafdc6d0ea8efdf08 (DEFAULT_TABLE table=test-table [id=c97b565e9b0d46e9a0871ad496bb58b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:10:57.012501 13555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fcd60b69247467fafdc6d0ea8efdf08. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:10:57.013872 13658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:10:56.996912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39949 (local address 127.0.0.1:60588)
0924 04:10:57.000440 (+ 3528us) negotiation.cc:107] Waiting for socket to connect
0924 04:10:57.000455 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:10:57.000547 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:10:57.006646 (+ 6099us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:10:57.006650 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:10:57.006669 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:10:57.006751 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:10:57.006757 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:10:57.008615 (+ 1858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:10:57.008618 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:10:57.009742 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:10:57.009752 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:10:57.013483 (+ 3731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:10:57.013487 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:10:57.013570 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:10:57.013593 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 04:10:57.013665 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 04:10:57.013735 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3465,"thread_start_us":78,"threads_started":1}
I20260924 04:10:57.015604 13656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:10:56.993824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33699 (local address 127.0.0.1:51700)
0924 04:10:56.998132 (+ 4308us) negotiation.cc:107] Waiting for socket to connect
0924 04:10:56.998183 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 04:10:56.998304 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:10:57.001821 (+ 3517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:10:57.001825 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:10:57.001857 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:10:57.001967 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:10:57.001973 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:10:57.011136 (+ 9163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:10:57.011139 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:10:57.012217 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:10:57.012227 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:10:57.015207 (+ 2980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:10:57.015211 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:10:57.015294 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:10:57.015339 (+ 45us) client_negotiation.cc:769] Sending connection context
0924 04:10:57.015411 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 04:10:57.015476 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4131,"thread_start_us":105,"threads_started":1}
I20260924 04:10:57.020354 13655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:10:56.993983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51700 (local address 127.0.98.2:33699)
0924 04:10:57.000812 (+ 6829us) server_negotiation.cc:206] Beginning negotiation
0924 04:10:57.000819 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 04:10:57.000842 (+ 23us) server_negotiation.cc:407] Connection header received
0924 04:10:57.000912 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:10:57.000928 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:10:57.001160 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:10:57.001365 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:10:57.009909 (+ 8544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:10:57.010997 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:10:57.014142 (+ 3145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:10:57.014905 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:10:57.015007 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:10:57.015134 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0924 04:10:57.020097 (+ 4963us) server_negotiation.cc:299] Negotiation successful
0924 04:10:57.020211 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6726,"thread_start_us":104,"threads_started":1}
I20260924 04:10:57.020848 13659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:10:56.997042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60588 (local address 127.0.98.1:39949)
0924 04:10:57.005966 (+ 8924us) server_negotiation.cc:206] Beginning negotiation
0924 04:10:57.005975 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 04:10:57.006002 (+ 27us) server_negotiation.cc:407] Connection header received
0924 04:10:57.006080 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:10:57.006085 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:10:57.006292 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:10:57.006488 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:10:57.007409 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:10:57.008491 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:10:57.012372 (+ 3881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:10:57.013168 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:10:57.013268 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:10:57.013418 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 04:10:57.020569 (+ 7151us) server_negotiation.cc:299] Negotiation successful
0924 04:10:57.020710 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8800,"thread_start_us":113,"threads_started":1}
I20260924 04:10:57.024142 13660 tablet_bootstrap.cc:492] T 8fcd60b69247467fafdc6d0ea8efdf08 P c7f4b9eb1354489f92a32f227eb109de: Bootstrap starting.
I20260924 04:10:57.024008 13424 tablet_service.cc:1467] Processing CreateTablet for tablet 8fcd60b69247467fafdc6d0ea8efdf08 (DEFAULT_TABLE table=test-table [id=c97b565e9b0d46e9a0871ad496bb58b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:10:57.024016 13290 tablet_service.cc:1467] Processing CreateTablet for tablet 8fcd60b69247467fafdc6d0ea8efdf08 (DEFAULT_TABLE table=test-table [id=c97b565e9b0d46e9a0871ad496bb58b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:10:57.025069 13290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fcd60b69247467fafdc6d0ea8efdf08. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:10:57.025360 13424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fcd60b69247467fafdc6d0ea8efdf08. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:10:57.026407 13660 tablet_bootstrap.cc:654] T 8fcd60b69247467fafdc6d0ea8efdf08 P c7f4b9eb1354489f92a32f227eb109de: Neither blocks nor log segments found. Creating new log.
I20260924 04:10:57.027487 13660 log.cc:824] T 8fcd60b69247467fafdc6d0ea8efdf08 P c7f4b9eb1354489f92a32f227eb109de: Log is configured to *not* fsync() on all Append() calls
I20260924 04:10:57.029611 13660 tablet_bootstrap.cc:492] T 8fcd60b69247467fafdc6d0ea8efdf08 P c7f4b9eb1354489f92a32f227eb109de: No bootstrap required, opened a new log
I20260924 04:10:57.029842 13660 ts_tablet_manager.cc:1397] T 8fcd60b69247467fafdc6d0ea8efdf08 P c7f4b9eb1354489f92a32f227eb109de: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 04:10:57.035367 13660 raft_consensus.cc:348] T 8fcd60b69247467fafdc6d0ea8efdf08 P c7f4b9eb1354489f92a32f227eb109de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36578d05718744b28d577898616a97e8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33699 } } peers { permanent_uuid: "c7f4b9eb1354489f92a32f227eb109de" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41761 } } peers { permanent_uuid: "21d1bee4a3ce411f8e564abfa332dcda" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39949 } }
I20260924 04:10:57.035826 13660 raft_consensus.cc:374] T 8fcd60b69247467fafdc6d0ea8efdf08 P c7f4b9eb1354489f92a32f227eb109de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:10:57.035959 13660 raft_consensus.cc:729] T 8fcd60b69247467fafdc6d0ea8efdf08 P c7f4b9eb1354489f92a32f227eb109de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7f4b9eb1354489f92a32f227eb109de, State: Initialized, Role: FOLLOWER
I20260924 04:10:57.036562 13660 consensus_queue.cc:260] T 8fcd60b69247467fafdc6d0ea8efdf08 P c7f4b9eb1354489f92a32f227eb109de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36578d05718744b28d577898616a97e8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33699 } } peers { permanent_uuid: "c7f4b9eb1354489f92a32f227eb109de" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41761 } } peers { permanent_uuid: "21d1bee4a3ce411f8e564abfa332dcda" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39949 } }
I20260924 04:10:57.038008 13660 ts_tablet_manager.cc:1428] T 8fcd60b69247467fafdc6d0ea8efdf08 P c7f4b9eb1354489f92a32f227eb109de: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260924 04:10:57.038416 13624 heartbeater.cc:503] Master 127.0.98.62:39255 was elected leader, sending a full tablet report...
I20260924 04:10:57.045017 13662 tablet_bootstrap.cc:492] T 8fcd60b69247467fafdc6d0ea8efdf08 P 21d1bee4a3ce411f8e564abfa332dcda: Bootstrap starting.
I20260924 04:10:57.047683 13662 tablet_bootstrap.cc:654] T 8fcd60b69247467fafdc6d0ea8efdf08 P 21d1bee4a3ce411f8e564abfa332dcda: Neither blocks nor log segments found. Creating new log.
I20260924 04:10:57.048830 13662 log.cc:824] T 8fcd60b69247467fafdc6d0ea8efdf08 P 21d1bee4a3ce411f8e564abfa332dcda: Log is configured to *not* fsync() on all Append() calls
I20260924 04:10:57.049561 13664 tablet_bootstrap.cc:492] T 8fcd60b69247467fafdc6d0ea8efdf08 P 36578d05718744b28d577898616a97e8: Bootstrap starting.
I20260924 04:10:57.051805 13664 tablet_bootstrap.cc:654] T 8fcd60b69247467fafdc6d0ea8efdf08 P 36578d05718744b28d577898616a97e8: Neither blocks nor log segments found. Creating new log.
I20260924 04:10:57.054936 13662 tablet_bootstrap.cc:492] T 8fcd60b69247467fafdc6d0ea8efdf08 P 21d1bee4a3ce411f8e564abfa332dcda: No bootstrap required, opened a new log
I20260924 04:10:57.055339 13662 ts_tablet_manager.cc:1397] T 8fcd60b69247467fafdc6d0ea8efdf08 P 21d1bee4a3ce411f8e564abfa332dcda: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260924 04:10:57.059813 13662 raft_consensus.cc:348] T 8fcd60b69247467fafdc6d0ea8efdf08 P 21d1bee4a3ce411f8e564abfa332dcda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36578d05718744b28d577898616a97e8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33699 } } peers { permanent_uuid: "c7f4b9eb1354489f92a32f227eb109de" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41761 } } peers { permanent_uuid: "21d1bee4a3ce411f8e564abfa332dcda" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39949 } }
I20260924 04:10:57.060253 13662 raft_consensus.cc:374] T 8fcd60b69247467fafdc6d0ea8efdf08 P 21d1bee4a3ce411f8e564abfa332dcda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:10:57.060361 13662 raft_consensus.cc:729] T 8fcd60b69247467fafdc6d0ea8efdf08 P 21d1bee4a3ce411f8e564abfa332dcda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21d1bee4a3ce411f8e564abfa332dcda, State: Initialized, Role: FOLLOWER
I20260924 04:10:57.061795 13664 log.cc:824] T 8fcd60b69247467fafdc6d0ea8efdf08 P 36578d05718744b28d577898616a97e8: Log is configured to *not* fsync() on all Append() calls
I20260924 04:10:57.062273 13662 consensus_queue.cc:260] T 8fcd60b69247467fafdc6d0ea8efdf08 P 21d1bee4a3ce411f8e564abfa332dcda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36578d05718744b28d577898616a97e8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33699 } } peers { permanent_uuid: "c7f4b9eb1354489f92a32f227eb109de" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41761 } } peers { permanent_uuid: "21d1bee4a3ce411f8e564abfa332dcda" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39949 } }
I20260924 04:10:57.063578 13662 ts_tablet_manager.cc:1428] T 8fcd60b69247467fafdc6d0ea8efdf08 P 21d1bee4a3ce411f8e564abfa332dcda: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260924 04:10:57.064407 13356 heartbeater.cc:503] Master 127.0.98.62:39255 was elected leader, sending a full tablet report...
I20260924 04:10:57.070581 13664 tablet_bootstrap.cc:492] T 8fcd60b69247467fafdc6d0ea8efdf08 P 36578d05718744b28d577898616a97e8: No bootstrap required, opened a new log
I20260924 04:10:57.070892 13664 ts_tablet_manager.cc:1397] T 8fcd60b69247467fafdc6d0ea8efdf08 P 36578d05718744b28d577898616a97e8: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.000s
I20260924 04:10:57.075425 13664 raft_consensus.cc:348] T 8fcd60b69247467fafdc6d0ea8efdf08 P 36578d05718744b28d577898616a97e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36578d05718744b28d577898616a97e8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33699 } } peers { permanent_uuid: "c7f4b9eb1354489f92a32f227eb109de" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41761 } } peers { permanent_uuid: "21d1bee4a3ce411f8e564abfa332dcda" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39949 } }
I20260924 04:10:57.077735 13664 raft_consensus.cc:374] T 8fcd60b69247467fafdc6d0ea8efdf08 P 36578d05718744b28d577898616a97e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:10:57.077860 13664 raft_consensus.cc:729] T 8fcd60b69247467fafdc6d0ea8efdf08 P 36578d05718744b28d577898616a97e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36578d05718744b28d577898616a97e8, State: Initialized, Role: FOLLOWER
I20260924 04:10:57.078425 13664 consensus_queue.cc:260] T 8fcd60b69247467fafdc6d0ea8efdf08 P 36578d05718744b28d577898616a97e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36578d05718744b28d577898616a97e8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33699 } } peers { permanent_uuid: "c7f4b9eb1354489f92a32f227eb109de" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41761 } } peers { permanent_uuid: "21d1bee4a3ce411f8e564abfa332dcda" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39949 } }
I20260924 04:10:57.079782 13664 ts_tablet_manager.cc:1428] T 8fcd60b69247467fafdc6d0ea8efdf08 P 36578d05718744b28d577898616a97e8: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260924 04:10:57.080186 13490 heartbeater.cc:503] Master 127.0.98.62:39255 was elected leader, sending a full tablet report...
I20260924 04:10:57.120501 13663 raft_consensus.cc:482] T 8fcd60b69247467fafdc6d0ea8efdf08 P c7f4b9eb1354489f92a32f227eb109de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:10:57.120865 13663 raft_consensus.cc:504] T 8fcd60b69247467fafdc6d0ea8efdf08 P c7f4b9eb1354489f92a32f227eb109de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36578d05718744b28d577898616a97e8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33699 } } peers { permanent_uuid: "c7f4b9eb1354489f92a32f227eb109de" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41761 } } peers { permanent_uuid: "21d1bee4a3ce411f8e564abfa332dcda" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39949 } }
I20260924 04:10:57.122264 13663 leader_election.cc:288] T 8fcd60b69247467fafdc6d0ea8efdf08 P c7f4b9eb1354489f92a32f227eb109de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 36578d05718744b28d577898616a97e8 (127.0.98.2:33699), 21d1bee4a3ce411f8e564abfa332dcda (127.0.98.1:39949)
W20260924 04:10:57.129101 13491 tablet.cc:2369] T 8fcd60b69247467fafdc6d0ea8efdf08 P 36578d05718744b28d577898616a97e8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:10:57.133006 13655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:10:57.123610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59603 (local address 127.0.98.2:33699)
0924 04:10:57.123766 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 04:10:57.123771 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:10:57.127551 (+ 3780us) server_negotiation.cc:407] Connection header received
0924 04:10:57.127759 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:10:57.127763 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:10:57.127816 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:10:57.127864 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:10:57.128291 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:10:57.129119 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:10:57.131134 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:10:57.131760 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:10:57.131838 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:10:57.131903 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 04:10:57.132798 (+ 895us) server_negotiation.cc:299] Negotiation successful
0924 04:10:57.132878 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 04:10:57.133656 13670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:10:57.122767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33699 (local address 127.0.98.3:59603)
0924 04:10:57.127441 (+ 4674us) negotiation.cc:107] Waiting for socket to connect
0924 04:10:57.127458 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 04:10:57.127637 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:10:57.128008 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:10:57.128011 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:10:57.128028 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:10:57.128169 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:10:57.128174 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:10:57.129263 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:10:57.129266 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:10:57.130625 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:10:57.130636 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:10:57.131995 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:10:57.131998 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:10:57.132108 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:10:57.132129 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 04:10:57.133417 (+ 1288us) client_negotiation.cc:240] Negotiation successful
0924 04:10:57.133495 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4595,"thread_start_us":144,"threads_started":1}
W20260924 04:10:57.135787 13357 tablet.cc:2369] T 8fcd60b69247467fafdc6d0ea8efdf08 P 21d1bee4a3ce411f8e564abfa332dcda: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:10:57.136852 13444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fcd60b69247467fafdc6d0ea8efdf08" candidate_uuid: "c7f4b9eb1354489f92a32f227eb109de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36578d05718744b28d577898616a97e8" is_pre_election: true
I20260924 04:10:57.137171 13444 raft_consensus.cc:2454] T 8fcd60b69247467fafdc6d0ea8efdf08 P 36578d05718744b28d577898616a97e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7f4b9eb1354489f92a32f227eb109de in term 0.
I20260924 04:10:57.137765 13514 leader_election.cc:302] T 8fcd60b69247467fafdc6d0ea8efdf08 P c7f4b9eb1354489f92a32f227eb109de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36578d05718744b28d577898616a97e8, c7f4b9eb1354489f92a32f227eb109de; no voters:
I20260924 04:10:57.138075 13663 raft_consensus.cc:2781] T 8fcd60b69247467fafdc6d0ea8efdf08 P c7f4b9eb1354489f92a32f227eb109de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:10:57.138142 13663 raft_consensus.cc:482] T 8fcd60b69247467fafdc6d0ea8efdf08 P c7f4b9eb1354489f92a32f227eb109de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:10:57.138177 13663 raft_consensus.cc:3037] T 8fcd60b69247467fafdc6d0ea8efdf08 P c7f4b9eb1354489f92a32f227eb109de [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:10:57.139521 13663 raft_consensus.cc:504] T 8fcd60b69247467fafdc6d0ea8efdf08 P c7f4b9eb1354489f92a32f227eb109de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36578d05718744b28d577898616a97e8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33699 } } peers { permanent_uuid: "c7f4b9eb1354489f92a32f227eb109de" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41761 } } peers { permanent_uuid: "21d1bee4a3ce411f8e564abfa332dcda" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39949 } }
I20260924 04:10:57.139879 13663 leader_election.cc:288] T 8fcd60b69247467fafdc6d0ea8efdf08 P c7f4b9eb1354489f92a32f227eb109de [CANDIDATE]: Term 1 election: Requested vote from peers 36578d05718744b28d577898616a97e8 (127.0.98.2:33699), 21d1bee4a3ce411f8e564abfa332dcda (127.0.98.1:39949)
I20260924 04:10:57.140461 13444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fcd60b69247467fafdc6d0ea8efdf08" candidate_uuid: "c7f4b9eb1354489f92a32f227eb109de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36578d05718744b28d577898616a97e8"
I20260924 04:10:57.140601 13444 raft_consensus.cc:3037] T 8fcd60b69247467fafdc6d0ea8efdf08 P 36578d05718744b28d577898616a97e8 [term 0 FOLLOWER]: Advancing to term 1
W20260924 04:10:57.141031 13625 tablet.cc:2369] T 8fcd60b69247467fafdc6d0ea8efdf08 P c7f4b9eb1354489f92a32f227eb109de: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:10:57.141837 13444 raft_consensus.cc:2454] T 8fcd60b69247467fafdc6d0ea8efdf08 P 36578d05718744b28d577898616a97e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7f4b9eb1354489f92a32f227eb109de in term 1.
I20260924 04:10:57.142215 13514 leader_election.cc:302] T 8fcd60b69247467fafdc6d0ea8efdf08 P c7f4b9eb1354489f92a32f227eb109de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36578d05718744b28d577898616a97e8, c7f4b9eb1354489f92a32f227eb109de; no voters:
I20260924 04:10:57.142454 13663 raft_consensus.cc:2781] T 8fcd60b69247467fafdc6d0ea8efdf08 P c7f4b9eb1354489f92a32f227eb109de [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:10:57.142654 13663 raft_consensus.cc:686] T 8fcd60b69247467fafdc6d0ea8efdf08 P c7f4b9eb1354489f92a32f227eb109de [term 1 LEADER]: Becoming Leader. State: Replica: c7f4b9eb1354489f92a32f227eb109de, State: Running, Role: LEADER
I20260924 04:10:57.142989 13663 consensus_queue.cc:237] T 8fcd60b69247467fafdc6d0ea8efdf08 P c7f4b9eb1354489f92a32f227eb109de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36578d05718744b28d577898616a97e8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33699 } } peers { permanent_uuid: "c7f4b9eb1354489f92a32f227eb109de" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41761 } } peers { permanent_uuid: "21d1bee4a3ce411f8e564abfa332dcda" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39949 } }
I20260924 04:10:57.146319 13157 catalog_manager.cc:5495] T 8fcd60b69247467fafdc6d0ea8efdf08 P c7f4b9eb1354489f92a32f227eb109de reported cstate change: term changed from 0 to 1, leader changed from <none> to c7f4b9eb1354489f92a32f227eb109de (127.0.98.3). New cstate: current_term: 1 leader_uuid: "c7f4b9eb1354489f92a32f227eb109de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36578d05718744b28d577898616a97e8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33699 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c7f4b9eb1354489f92a32f227eb109de" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21d1bee4a3ce411f8e564abfa332dcda" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39949 } health_report { overall_health: UNKNOWN } } }
I20260924 04:10:57.149953 13659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:10:57.123750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41657 (local address 127.0.98.1:39949)
0924 04:10:57.123879 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 04:10:57.123883 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:10:57.130964 (+ 7081us) server_negotiation.cc:407] Connection header received
0924 04:10:57.132261 (+ 1297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:10:57.132265 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:10:57.132356 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:10:57.132406 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:10:57.133929 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:10:57.134720 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:10:57.148519 (+ 13799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:10:57.149221 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:10:57.149323 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:10:57.149392 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 04:10:57.149747 (+ 355us) server_negotiation.cc:299] Negotiation successful
0924 04:10:57.149830 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 04:10:57.150507 13669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:10:57.123188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39949 (local address 127.0.98.3:41657)
0924 04:10:57.130881 (+ 7693us) negotiation.cc:107] Waiting for socket to connect
0924 04:10:57.130895 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:10:57.131030 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:10:57.132532 (+ 1502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:10:57.132535 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:10:57.132551 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:10:57.132654 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:10:57.132659 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:10:57.134861 (+ 2202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:10:57.134865 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:10:57.136085 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:10:57.136095 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:10:57.149496 (+ 13401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:10:57.149499 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:10:57.149611 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:10:57.149633 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 04:10:57.150265 (+ 632us) client_negotiation.cc:240] Negotiation successful
0924 04:10:57.150365 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7628,"thread_start_us":81,"threads_started":1}
I20260924 04:10:57.151238 13302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fcd60b69247467fafdc6d0ea8efdf08" candidate_uuid: "c7f4b9eb1354489f92a32f227eb109de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21d1bee4a3ce411f8e564abfa332dcda" is_pre_election: true
I20260924 04:10:57.151561 13302 raft_consensus.cc:2454] T 8fcd60b69247467fafdc6d0ea8efdf08 P 21d1bee4a3ce411f8e564abfa332dcda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7f4b9eb1354489f92a32f227eb109de in term 0.
I20260924 04:10:57.151733 13303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fcd60b69247467fafdc6d0ea8efdf08" candidate_uuid: "c7f4b9eb1354489f92a32f227eb109de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21d1bee4a3ce411f8e564abfa332dcda"
I20260924 04:10:57.151855 13303 raft_consensus.cc:3037] T 8fcd60b69247467fafdc6d0ea8efdf08 P 21d1bee4a3ce411f8e564abfa332dcda [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:10:57.153178 13303 raft_consensus.cc:2454] T 8fcd60b69247467fafdc6d0ea8efdf08 P 21d1bee4a3ce411f8e564abfa332dcda [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7f4b9eb1354489f92a32f227eb109de in term 1.
I20260924 04:10:57.167763 13657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:10:57.163165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37622 (local address 127.0.98.3:41761)
0924 04:10:57.163286 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0924 04:10:57.163291 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:10:57.163305 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:10:57.163387 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:10:57.163390 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:10:57.163456 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:10:57.163562 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:10:57.163954 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:10:57.164754 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:10:57.166026 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:10:57.166378 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:10:57.166490 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:10:57.166927 (+ 437us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:10:57.167355 (+ 428us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:10:57.167497 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0924 04:10:57.167537 (+ 40us) server_negotiation.cc:299] Negotiation successful
0924 04:10:57.167592 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 04:10:57.173638 13303 raft_consensus.cc:1261] T 8fcd60b69247467fafdc6d0ea8efdf08 P 21d1bee4a3ce411f8e564abfa332dcda [term 1 FOLLOWER]: Refusing update from remote peer c7f4b9eb1354489f92a32f227eb109de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:10:57.173782 13444 raft_consensus.cc:1261] T 8fcd60b69247467fafdc6d0ea8efdf08 P 36578d05718744b28d577898616a97e8 [term 1 FOLLOWER]: Refusing update from remote peer c7f4b9eb1354489f92a32f227eb109de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:10:57.174455 13671 consensus_queue.cc:1035] T 8fcd60b69247467fafdc6d0ea8efdf08 P c7f4b9eb1354489f92a32f227eb109de [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d1bee4a3ce411f8e564abfa332dcda" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:10:57.174602 13663 consensus_queue.cc:1035] T 8fcd60b69247467fafdc6d0ea8efdf08 P c7f4b9eb1354489f92a32f227eb109de [LEADER]: Connected to new peer: Peer: permanent_uuid: "36578d05718744b28d577898616a97e8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:10:57.183574 13683 mvcc.cc:205] Tried to move back new op lower bound from 7332753642175791104 to 7332753642060951552. Current Snapshot: MvccSnapshot[applied={T|T < 7332753642175791104}]
I20260924 04:10:57.183807 13681 mvcc.cc:205] Tried to move back new op lower bound from 7332753642175791104 to 7332753642060951552. Current Snapshot: MvccSnapshot[applied={T|T < 7332753642175791104}]
I20260924 04:10:57.191972 13682 mvcc.cc:205] Tried to move back new op lower bound from 7332753642175791104 to 7332753642060951552. Current Snapshot: MvccSnapshot[applied={T|T < 7332753642175791104}]
I20260924 04:10:57.195005 13158 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47630:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:10:57.195175 13158 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47630:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:10:57.198094 13158 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ce511fd242bb4e408d08181ef52e1b32: Sending DeleteTablet for 3 replicas of tablet 8fcd60b69247467fafdc6d0ea8efdf08
I20260924 04:10:57.199033 13555 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fcd60b69247467fafdc6d0ea8efdf08 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:10:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37614
I20260924 04:10:57.200488 13290 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fcd60b69247467fafdc6d0ea8efdf08 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:10:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60588
I20260924 04:10:57.200618 13424 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fcd60b69247467fafdc6d0ea8efdf08 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:10:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51700
I20260924 04:10:57.201676 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 13196
I20260924 04:10:57.205904 13690 tablet_replica.cc:330] T 8fcd60b69247467fafdc6d0ea8efdf08 P c7f4b9eb1354489f92a32f227eb109de: stopping tablet replica
I20260924 04:10:57.206269 13690 raft_consensus.cc:2229] T 8fcd60b69247467fafdc6d0ea8efdf08 P c7f4b9eb1354489f92a32f227eb109de [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:10:57.206516 13692 tablet_replica.cc:330] T 8fcd60b69247467fafdc6d0ea8efdf08 P 36578d05718744b28d577898616a97e8: stopping tablet replica
I20260924 04:10:57.206916 13690 raft_consensus.cc:2258] T 8fcd60b69247467fafdc6d0ea8efdf08 P c7f4b9eb1354489f92a32f227eb109de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:10:57.207264 13692 raft_consensus.cc:2229] T 8fcd60b69247467fafdc6d0ea8efdf08 P 36578d05718744b28d577898616a97e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:10:57.207720 13692 raft_consensus.cc:2258] T 8fcd60b69247467fafdc6d0ea8efdf08 P 36578d05718744b28d577898616a97e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:10:57.210479 13690 ts_tablet_manager.cc:1905] T 8fcd60b69247467fafdc6d0ea8efdf08 P c7f4b9eb1354489f92a32f227eb109de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:10:57.213359 13690 ts_tablet_manager.cc:1918] T 8fcd60b69247467fafdc6d0ea8efdf08 P c7f4b9eb1354489f92a32f227eb109de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:10:57.215606 13692 ts_tablet_manager.cc:1905] T 8fcd60b69247467fafdc6d0ea8efdf08 P 36578d05718744b28d577898616a97e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:10:57.218182 13692 ts_tablet_manager.cc:1918] T 8fcd60b69247467fafdc6d0ea8efdf08 P 36578d05718744b28d577898616a97e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:10:57.218359 13690 log.cc:1196] T 8fcd60b69247467fafdc6d0ea8efdf08 P c7f4b9eb1354489f92a32f227eb109de: Deleting WAL directory at /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ts-2/wal/wals/8fcd60b69247467fafdc6d0ea8efdf08
I20260924 04:10:57.218732 13690 ts_tablet_manager.cc:1939] T 8fcd60b69247467fafdc6d0ea8efdf08 P c7f4b9eb1354489f92a32f227eb109de: Deleting consensus metadata
I20260924 04:10:57.218880 13692 log.cc:1196] T 8fcd60b69247467fafdc6d0ea8efdf08 P 36578d05718744b28d577898616a97e8: Deleting WAL directory at /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790222898807720-392-0/minicluster-data/ts-1/wal/wals/8fcd60b69247467fafdc6d0ea8efdf08
W20260924 04:10:57.219980 13145 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39949 (error 108)
I20260924 04:10:57.220160 13142 catalog_manager.cc:4841] TS c7f4b9eb1354489f92a32f227eb109de (127.0.98.3:41761): tablet 8fcd60b69247467fafdc6d0ea8efdf08 (table test-table [id=c97b565e9b0d46e9a0871ad496bb58b5]) successfully deleted
I20260924 04:10:57.220674 13692 ts_tablet_manager.cc:1939] T 8fcd60b69247467fafdc6d0ea8efdf08 P 36578d05718744b28d577898616a97e8: Deleting consensus metadata
I20260924 04:10:57.221858 13145 catalog_manager.cc:4841] TS 36578d05718744b28d577898616a97e8 (127.0.98.2:33699): tablet 8fcd60b69247467fafdc6d0ea8efdf08 (table test-table [id=c97b565e9b0d46e9a0871ad496bb58b5]) successfully deleted
I20260924 04:10:57.222322 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 13358
I20260924 04:10:57.224433 13654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:10:57.224121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39949 (local address 127.0.0.1:60592)
0924 04:10:57.224285 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 04:10:57.224350 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
W20260924 04:10:57.224655 13145 catalog_manager.cc:4584] TS 21d1bee4a3ce411f8e564abfa332dcda (127.0.98.1:39949): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8fcd60b69247467fafdc6d0ea8efdf08: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39949: connect: Connection refused (error 111)
I20260924 04:10:57.229382 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 13493
I20260924 04:10:57.236449 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 13125
I20260924 04:10:57.243805 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 04:10:57.588204 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 04:10:57.588305 392 mini_ranger.cc:66] Stopping Ranger...
I20260924 04:10:57.933354 392 mini_ranger.cc:68] Stopped Ranger
2026-09-24 04:10:57.933 UTC [12192] LOG: received smart shutdown request
2026-09-24 04:10:57.937 UTC [12192] LOG: background worker "logical replication launcher" (PID 12201) exited with exit code 1
2026-09-24 04:10:57.937 UTC [12196] LOG: shutting down
2026-09-24 04:10:58.115 UTC [12192] LOG: database system is shut down
2026-09-24T04:10:58Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (48674 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222898807720-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:10:58 dist-test-slave-00ld krb5kdc[13702](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:10:58 dist-test-slave-00ld krb5kdc[13702](info): set up 2 sockets
Sep 24 04:10:58 dist-test-slave-00ld krb5kdc[13702](info): commencing operation
krb5kdc: starting...
W20260924 04:10:59.308869 392 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222898807720-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222898807720-392-0/krb5kdc/test-user.keytab.
Sep 24 04:10:59 dist-test-slave-00ld krb5kdc[13702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223059, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:10:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:10:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:10:59.436376 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222898807720-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222898807720-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222898807720-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39917
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39917,127.0.98.61:44899,127.0.98.60:37463
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:10:59.548849 13715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:10:59.549121 13715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:10:59.549163 13715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:10:59.553087 13715 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:10:59.553141 13715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:10:59.553161 13715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:10:59.553177 13715 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:10:59.553193 13715 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:10:59.558403 13715 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222898807720-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222898807720-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39917,127.0.98.61:44899,127.0.98.60:37463
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39917
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13715
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222898807720-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:10:59.559453 13715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:10:59.560448 13715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:10:59.568949 13723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:10:59.568986 13720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:10:59.569025 13721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:10:59.569350 13715 server_base.cc:1034] running on GCE node
I20260924 04:10:59.569918 13715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:10:59.571014 13715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:10:59.572213 13715 hybrid_clock.cc:648] HybridClock initialized: now 1790223059572200 us; error 26 us; skew 500 ppm
Sep 24 04:10:59 dist-test-slave-00ld krb5kdc[13702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223059, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:10:59.575075 13715 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:10:59.575928 13715 webserver.cc:460] Webserver started at http://127.0.98.62:32901/ using document root <none> and password file <none>
I20260924 04:10:59.576483 13715 fs_manager.cc:359] Metadata directory not provided
I20260924 04:10:59.576591 13715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:10:59.576858 13715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:10:59.578888 13715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222898807720-392-0/minicluster-data/master-0/data/instance:
uuid: "9485a46effd14f048c7046ae9da7bb92"
format_stamp: "Formatted at 2026-09-24 04:10:59 on dist-test-slave-00ld"
server_key: "39960f8f089fd96ec6737a71c9dde0c4"
server_key_iv: "59275ac11efb0d14b344ef361614eee1"
server_key_version: "encryptionkey@0"
I20260924 04:10:59.579442 13715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222898807720-392-0/minicluster-data/master-0/wal/instance:
uuid: "9485a46effd14f048c7046ae9da7bb92"
format_stamp: "Formatted at 2026-09-24 04:10:59 on dist-test-slave-00ld"
server_key: "39960f8f089fd96ec6737a71c9dde0c4"
server_key_iv: "59275ac11efb0d14b344ef361614eee1"
server_key_version: "encryptionkey@0"
I20260924 04:10:59.583117 13715 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 04:10:59.585637 13730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:10:59.586915 13715 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:10:59.587055 13715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222898807720-392-0/minicluster-data/master-0/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222898807720-392-0/minicluster-data/master-0/wal
uuid: "9485a46effd14f048c7046ae9da7bb92"
format_stamp: "Formatted at 2026-09-24 04:10:59 on dist-test-slave-00ld"
server_key: "39960f8f089fd96ec6737a71c9dde0c4"
server_key_iv: "59275ac11efb0d14b344ef361614eee1"
server_key_version: "encryptionkey@0"
I20260924 04:10:59.587168 13715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222898807720-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222898807720-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222898807720-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:10:59.606382 13715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:10:59.609432 13715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:10:59.609642 13715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:10:59.616815 13715 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39917
I20260924 04:10:59.616842 13782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39917 every 8 connection(s)
I20260924 04:10:59.617758 13715 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:10:59.620450 13783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:10:59.623021 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 13715
I20260924 04:10:59.623131 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222898807720-392-0/minicluster-data/master-0/wal/instance
I20260924 04:10:59.623428 392 external_mini_cluster.cc:1428] Setting key 13bc25a522b5f344ec59505be3f7caee
I20260924 04:10:59.624269 13783 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39917 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 24 04:10:59 dist-test-slave-00ld krb5kdc[13702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223059, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:10:59.640180 13787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:10:59.626662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39917 (local address 127.0.0.1:60144)
0924 04:10:59.627285 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0924 04:10:59.627333 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 04:10:59.628163 (+ 830us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:10:59.629023 (+ 860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:10:59.629027 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:10:59.629347 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:10:59.629628 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:10:59.629641 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:10:59.631335 (+ 1694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:10:59.631338 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:10:59.632123 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:10:59.632132 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:10:59.632849 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:10:59.632852 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:10:59.632951 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:10:59.633530 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:10:59.633546 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:10:59.635017 (+ 1471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:10:59.637225 (+ 2208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:10:59.637234 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:10:59.637244 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:10:59.637466 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:10:59.637784 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:10:59.637787 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:10:59.637789 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:10:59.637903 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:10:59.638486 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:10:59.638492 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:10:59.638629 (+ 137us) client_negotiation.cc:769] Sending connection context
0924 04:10:59.638791 (+ 162us) client_negotiation.cc:240] Negotiation successful
0924 04:10:59.639161 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"mutex_wait_us":104,"spinlock_wait_cycles":55552,"thread_start_us":162,"threads_started":1}
I20260924 04:10:59.640183 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:10:59.626662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60144 (local address 127.0.98.62:39917)
0924 04:10:59.627250 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0924 04:10:59.627272 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0924 04:10:59.627465 (+ 193us) server_negotiation.cc:407] Connection header received
0924 04:10:59.628482 (+ 1017us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:10:59.628491 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:10:59.628803 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:10:59.629023 (+ 220us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:10:59.629780 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:10:59.631206 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:10:59.632231 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:10:59.632641 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:10:59.632823 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:10:59.635170 (+ 2347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:10:59.635176 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:10:59.635185 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:10:59.635204 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:10:59.637122 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:10:59.637574 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:10:59.637581 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:10:59.637588 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:10:59.637684 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:10:59.637990 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:10:59.637996 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:10:59.637998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:10:59.638360 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:10:59.638504 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0924 04:10:59.638860 (+ 356us) server_negotiation.cc:299] Negotiation successful
0924 04:10:59.639110 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":162,"threads_started":1}
I20260924 04:10:59.642709 13783 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44899 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:10:59.643461 13787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:10:59.643169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44899 (local address 127.0.0.1:58132)
0924 04:10:59.643324 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 04:10:59.643391 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260924 04:10:59.643671 13732 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44899: connect: Connection refused (error 111)
I20260924 04:10:59.644732 13787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:10:59.644435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44899 (local address 127.0.0.1:58148)
0924 04:10:59.644549 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0924 04:10:59.644629 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260924 04:10:59.644891 13783 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:44899: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44899: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222898807720-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222898807720-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260924 04:10:59.651883 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222898807720-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222898807720-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222898807720-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222898807720-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:44899
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222898807720-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39917,127.0.98.61:44899,127.0.98.60:37463
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:10:59.694173 13783 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44899 } attempt: 1
I20260924 04:10:59.695185 13787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:10:59.694873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44899 (local address 127.0.0.1:58160)
0924 04:10:59.695039 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 04:10:59.695107 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260924 04:10:59.696009 13787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:10:59.695801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44899 (local address 127.0.0.1:58170)
0924 04:10:59.695907 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0924 04:10:59.695954 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260924 04:10:59.696190 13783 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:44899: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44899: connect: Connection refused (error 111)
W20260924 04:10:59.763221 13790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:10:59.763497 13790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:10:59.763537 13790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260924 04:10:59.764385 13783 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44899 } attempt: 2
I20260924 04:10:59.765414 13787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:10:59.764998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44899 (local address 127.0.0.1:58178)
0924 04:10:59.765171 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 04:10:59.765246 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260924 04:10:59.766237 13787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:10:59.765999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44899 (local address 127.0.0.1:58184)
0924 04:10:59.766126 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 04:10:59.766177 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260924 04:10:59.766397 13783 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:44899: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44899: connect: Connection refused (error 111)
W20260924 04:10:59.768308 13790 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:10:59.768359 13790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:10:59.768378 13790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:10:59.768404 13790 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:10:59.768460 13790 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:10:59.773968 13790 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222898807720-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222898807720-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39917,127.0.98.61:44899,127.0.98.60:37463
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222898807720-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:44899
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222898807720-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13790
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222898807720-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:10:59.775043 13790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:10:59.776082 13790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:10:59.784621 13796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:10:59.784632 13795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:10:59.784619 13798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:10:59.784934 13790 server_base.cc:1034] running on GCE node
I20260924 04:10:59.785547 13790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:10:59.786562 13790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:10:59.787750 13790 hybrid_clock.cc:648] HybridClock initialized: now 1790223059787717 us; error 45 us; skew 500 ppm
Sep 24 04:10:59 dist-test-slave-00ld krb5kdc[13702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223059, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:10:59.790653 13790 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260924 04:10:59.791625 13790 webserver.cc:460] Webserver started at http://127.0.98.61:38347/ using document root <none> and password file <none>
I20260924 04:10:59.792153 13790 fs_manager.cc:359] Metadata directory not provided
I20260924 04:10:59.792212 13790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:10:59.792382 13790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:10:59.794400 13790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222898807720-392-0/minicluster-data/master-1/data/instance:
uuid: "b8b0bf0e2e534ce19ce3934bd892bab2"
format_stamp: "Formatted at 2026-09-24 04:10:59 on dist-test-slave-00ld"
server_key: "c919069d09edb45087236b84c08235c9"
server_key_iv: "95b1e9a6b0ce70eac4f896cf72eb9f95"
server_key_version: "encryptionkey@0"
I20260924 04:10:59.794899 13790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222898807720-392-0/minicluster-data/master-1/wal/instance:
uuid: "b8b0bf0e2e534ce19ce3934bd892bab2"
format_stamp: "Formatted at 2026-09-24 04:10:59 on dist-test-slave-00ld"
server_key: "c919069d09edb45087236b84c08235c9"
server_key_iv: "95b1e9a6b0ce70eac4f896cf72eb9f95"
server_key_version: "encryptionkey@0"
I20260924 04:10:59.798792 13790 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260924 04:10:59.801479 13805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:10:59.802661 13790 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:10:59.802762 13790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222898807720-392-0/minicluster-data/master-1/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222898807720-392-0/minicluster-data/master-1/wal
uuid: "b8b0bf0e2e534ce19ce3934bd892bab2"
format_stamp: "Formatted at 2026-09-24 04:10:59 on dist-test-slave-00ld"
server_key: "c919069d09edb45087236b84c08235c9"
server_key_iv: "95b1e9a6b0ce70eac4f896cf72eb9f95"
server_key_version: "encryptionkey@0"
I20260924 04:10:59.802845 13790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222898807720-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222898807720-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222898807720-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:10:59.821614 13790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:10:59.824585 13790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:10:59.824811 13790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:10:59.831734 13790 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:44899
I20260924 04:10:59.831818 13857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:44899 every 8 connection(s)
I20260924 04:10:59.832841 13790 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222898807720-392-0/minicluster-data/master-1/data/info.pb
I20260924 04:10:59.835988 13858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:10:59.838323 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 13790
I20260924 04:10:59.838472 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222898807720-392-0/minicluster-data/master-1/wal/instance
I20260924 04:10:59.838788 392 external_mini_cluster.cc:1428] Setting key e3332cb723c79e7aad0941aeeaa81fe3
I20260924 04:10:59.840834 13858 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39917 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 24 04:10:59 dist-test-slave-00ld krb5kdc[13702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223059, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:10:59.855587 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:10:59.842869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60158 (local address 127.0.98.62:39917)
0924 04:10:59.843050 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 04:10:59.843055 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:10:59.843997 (+ 942us) server_negotiation.cc:407] Connection header received
0924 04:10:59.845055 (+ 1058us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:10:59.845061 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:10:59.845138 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:10:59.845251 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:10:59.846442 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:10:59.847241 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:10:59.848648 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:10:59.849023 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:10:59.849120 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:10:59.851883 (+ 2763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:10:59.851892 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:10:59.851894 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:10:59.851920 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:10:59.853500 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:10:59.854006 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:10:59.854016 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:10:59.854018 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:10:59.854105 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:10:59.854447 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:10:59.854452 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:10:59.854453 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:10:59.854646 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:10:59.854751 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 04:10:59.855260 (+ 509us) server_negotiation.cc:299] Negotiation successful
0924 04:10:59.855410 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 04:10:59.856390 13861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:10:59.843292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39917 (local address 127.0.0.1:60158)
0924 04:10:59.843805 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0924 04:10:59.843859 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 04:10:59.844824 (+ 965us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:10:59.845410 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:10:59.845417 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:10:59.845756 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:10:59.846269 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:10:59.846283 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:10:59.847427 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:10:59.847430 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:10:59.848468 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:10:59.848478 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:10:59.849185 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:10:59.849189 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:10:59.849343 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:10:59.850053 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:10:59.850070 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:10:59.851706 (+ 1636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:10:59.853616 (+ 1910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:10:59.853622 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:10:59.853639 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:10:59.853900 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:10:59.854204 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:10:59.854207 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:10:59.854209 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:10:59.854353 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:10:59.854750 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:10:59.854758 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:10:59.855025 (+ 267us) client_negotiation.cc:769] Sending connection context
0924 04:10:59.855219 (+ 194us) client_negotiation.cc:240] Negotiation successful
0924 04:10:59.855570 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":161,"threads_started":1}
I20260924 04:10:59.857574 13783 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44899 } attempt: 3
I20260924 04:10:59.857878 13858 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44899 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 04:10:59 dist-test-slave-00ld krb5kdc[13702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223059, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 24 04:10:59 dist-test-slave-00ld krb5kdc[13702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223059, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222898807720-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222898807720-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260924 04:10:59.866477 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222898807720-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222898807720-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222898807720-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222898807720-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:37463
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222898807720-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39917,127.0.98.61:44899,127.0.98.60:37463
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:10:59.869336 13861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:10:59.858393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44899 (local address 127.0.0.1:58194)
0924 04:10:59.858481 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0924 04:10:59.858493 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:10:59.858568 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:10:59.859040 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:10:59.859045 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:10:59.859200 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:10:59.859318 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:10:59.859329 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:10:59.861120 (+ 1791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:10:59.861125 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:10:59.862065 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:10:59.862076 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:10:59.862723 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:10:59.862726 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:10:59.862836 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:10:59.863344 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:10:59.863392 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:10:59.864837 (+ 1445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:10:59.867722 (+ 2885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:10:59.867728 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:10:59.867733 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:10:59.868000 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:10:59.868261 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:10:59.868264 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:10:59.868266 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:10:59.868349 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:10:59.868820 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:10:59.868822 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:10:59.868912 (+ 90us) client_negotiation.cc:769] Sending connection context
0924 04:10:59.869020 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 04:10:59.869152 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":20}
I20260924 04:10:59.869395 13787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:10:59.857984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44899 (local address 127.0.0.1:58192)
0924 04:10:59.858148 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 04:10:59.858169 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 04:10:59.858306 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:10:59.858841 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:10:59.858843 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:10:59.859117 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:10:59.859250 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:10:59.859256 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:10:59.861573 (+ 2317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:10:59.861577 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:10:59.862417 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:10:59.862426 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:10:59.863037 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:10:59.863041 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:10:59.863129 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:10:59.863453 (+ 324us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:10:59.863471 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:10:59.865304 (+ 1833us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:10:59.867421 (+ 2117us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:10:59.867426 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:10:59.867428 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:10:59.867628 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:10:59.867998 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:10:59.868003 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:10:59.868006 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:10:59.868104 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:10:59.868797 (+ 693us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:10:59.868800 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:10:59.868915 (+ 115us) client_negotiation.cc:769] Sending connection context
0924 04:10:59.869008 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 04:10:59.869165 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100}
I20260924 04:10:59.869663 13864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:10:59.858574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58194 (local address 127.0.98.61:44899)
0924 04:10:59.858837 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0924 04:10:59.858850 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 04:10:59.858861 (+ 11us) server_negotiation.cc:407] Connection header received
0924 04:10:59.858886 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:10:59.858890 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:10:59.858923 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:10:59.859004 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:10:59.859421 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:10:59.861007 (+ 1586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:10:59.862174 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:10:59.862615 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:10:59.862712 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:10:59.864973 (+ 2261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:10:59.864983 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:10:59.865000 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:10:59.865024 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:10:59.867606 (+ 2582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:10:59.868099 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:10:59.868101 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:10:59.868102 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:10:59.868161 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:10:59.868441 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:10:59.868446 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:10:59.868448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:10:59.868631 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:10:59.868820 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0924 04:10:59.869128 (+ 308us) server_negotiation.cc:299] Negotiation successful
0924 04:10:59.869257 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":198,"thread_start_us":84,"threads_started":1}
I20260924 04:10:59.870191 13863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:10:59.858072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58192 (local address 127.0.98.61:44899)
0924 04:10:59.858450 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0924 04:10:59.858462 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 04:10:59.858484 (+ 22us) server_negotiation.cc:407] Connection header received
0924 04:10:59.858533 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:10:59.858541 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:10:59.858731 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:10:59.858860 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:10:59.859365 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:10:59.861006 (+ 1641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:10:59.862572 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:10:59.862903 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:10:59.862973 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:10:59.865476 (+ 2503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:10:59.865482 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:10:59.865488 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:10:59.865530 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:10:59.867301 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:10:59.867739 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:10:59.867745 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:10:59.867747 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:10:59.867873 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:10:59.868225 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:10:59.868229 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:10:59.868232 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:10:59.868627 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:10:59.869715 (+ 1088us) server_negotiation.cc:1035] Waiting for connection context
0924 04:10:59.869795 (+ 80us) server_negotiation.cc:299] Negotiation successful
0924 04:10:59.869922 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":347,"server-negotiator.queue_time_us":283,"spinlock_wait_cycles":4352,"thread_start_us":172,"threads_started":1}
I20260924 04:10:59.871699 13858 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37463 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:10:59.871860 13783 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37463 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:10:59.872417 13861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:10:59.872157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37463 (local address 127.0.0.1:53760)
0924 04:10:59.872272 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0924 04:10:59.872350 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260924 04:10:59.872589 13787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:10:59.872268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37463 (local address 127.0.0.1:53770)
0924 04:10:59.872470 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0924 04:10:59.872525 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
W20260924 04:10:59.872592 13808 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37463: connect: Connection refused (error 111)
I20260924 04:10:59.873376 13787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:10:59.873070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37463 (local address 127.0.0.1:53772)
0924 04:10:59.873249 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0924 04:10:59.873288 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
I20260924 04:10:59.873513 13861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:10:59.873279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37463 (local address 127.0.0.1:53786)
0924 04:10:59.873399 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0924 04:10:59.873451 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260924 04:10:59.873541 13783 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37463: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37463: connect: Connection refused (error 111)
W20260924 04:10:59.873623 13858 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37463: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37463: connect: Connection refused (error 111)
I20260924 04:10:59.904726 13783 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37463 } attempt: 1
I20260924 04:10:59.905598 13787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:10:59.905296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37463 (local address 127.0.0.1:53800)
0924 04:10:59.905445 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 04:10:59.905521 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260924 04:10:59.906352 13787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:10:59.906130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37463 (local address 127.0.0.1:53814)
0924 04:10:59.906241 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0924 04:10:59.906295 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260924 04:10:59.906533 13783 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37463: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37463: connect: Connection refused (error 111)
I20260924 04:10:59.922886 13858 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37463 } attempt: 1
I20260924 04:10:59.923714 13861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:10:59.923436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37463 (local address 127.0.0.1:53818)
0924 04:10:59.923577 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 04:10:59.923637 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260924 04:10:59.924497 13861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:10:59.924233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37463 (local address 127.0.0.1:53834)
0924 04:10:59.924345 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 04:10:59.924438 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260924 04:10:59.924638 13858 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37463: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37463: connect: Connection refused (error 111)
W20260924 04:10:59.980078 13865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:10:59.980399 13865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:10:59.980484 13865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260924 04:10:59.981703 13783 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37463 } attempt: 2
I20260924 04:10:59.982517 13787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:10:59.982207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37463 (local address 127.0.0.1:53844)
0924 04:10:59.982351 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 04:10:59.982418 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260924 04:10:59.983196 13787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:10:59.982986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37463 (local address 127.0.0.1:53856)
0924 04:10:59.983089 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0924 04:10:59.983134 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260924 04:10:59.983333 13783 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37463: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37463: connect: Connection refused (error 111)
W20260924 04:10:59.984869 13865 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:10:59.984916 13865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:10:59.984935 13865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:10:59.984975 13865 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:10:59.985019 13865 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:10:59.990394 13865 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222898807720-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222898807720-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39917,127.0.98.61:44899,127.0.98.60:37463
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222898807720-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:37463
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222898807720-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13865
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222898807720-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:10:59.991520 13865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:10:59.992609 13865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:10:59.992836 13858 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37463 } attempt: 2
I20260924 04:10:59.993618 13861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:10:59.993303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37463 (local address 127.0.0.1:53860)
0924 04:10:59.993491 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0924 04:10:59.993544 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260924 04:10:59.994293 13861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:10:59.994087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37463 (local address 127.0.0.1:53874)
0924 04:10:59.994190 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0924 04:10:59.994240 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260924 04:10:59.994417 13858 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37463: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37463: connect: Connection refused (error 111)
W20260924 04:11:00.001451 13871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:11:00.001451 13872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:11:00.001576 13874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:11:00.001631 13865 server_base.cc:1034] running on GCE node
I20260924 04:11:00.002279 13865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:11:00.003413 13865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:11:00.004603 13865 hybrid_clock.cc:648] HybridClock initialized: now 1790223060004586 us; error 32 us; skew 500 ppm
Sep 24 04:11:00 dist-test-slave-00ld krb5kdc[13702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223060, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:11:00.007663 13865 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260924 04:11:00.008575 13865 webserver.cc:460] Webserver started at http://127.0.98.60:35311/ using document root <none> and password file <none>
I20260924 04:11:00.009153 13865 fs_manager.cc:359] Metadata directory not provided
I20260924 04:11:00.009214 13865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:11:00.009378 13865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:11:00.011346 13865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222898807720-392-0/minicluster-data/master-2/data/instance:
uuid: "d62f8761fba24465beb67ac51480840e"
format_stamp: "Formatted at 2026-09-24 04:11:00 on dist-test-slave-00ld"
server_key: "b0066e5b63b024ed34bd12478e996f42"
server_key_iv: "81aff52c9f580f20acfcf22773383252"
server_key_version: "encryptionkey@0"
I20260924 04:11:00.011866 13865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222898807720-392-0/minicluster-data/master-2/wal/instance:
uuid: "d62f8761fba24465beb67ac51480840e"
format_stamp: "Formatted at 2026-09-24 04:11:00 on dist-test-slave-00ld"
server_key: "b0066e5b63b024ed34bd12478e996f42"
server_key_iv: "81aff52c9f580f20acfcf22773383252"
server_key_version: "encryptionkey@0"
I20260924 04:11:00.015581 13865 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260924 04:11:00.018230 13881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:11:00.019397 13865 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:11:00.019523 13865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222898807720-392-0/minicluster-data/master-2/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222898807720-392-0/minicluster-data/master-2/wal
uuid: "d62f8761fba24465beb67ac51480840e"
format_stamp: "Formatted at 2026-09-24 04:11:00 on dist-test-slave-00ld"
server_key: "b0066e5b63b024ed34bd12478e996f42"
server_key_iv: "81aff52c9f580f20acfcf22773383252"
server_key_version: "encryptionkey@0"
I20260924 04:11:00.019635 13865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222898807720-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222898807720-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222898807720-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:11:00.039966 13865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:11:00.043025 13865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:11:00.043221 13865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:11:00.050416 13865 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:37463
I20260924 04:11:00.050443 13933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:37463 every 8 connection(s)
I20260924 04:11:00.051335 13865 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222898807720-392-0/minicluster-data/master-2/data/info.pb
I20260924 04:11:00.053830 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 13865
I20260924 04:11:00.053989 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790222898807720-392-0/minicluster-data/master-2/wal/instance
I20260924 04:11:00.054243 13934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:11:00.054312 392 external_mini_cluster.cc:1428] Setting key 9a2c4471499a0ec71e97386da4b34568
I20260924 04:11:00.058758 13934 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39917 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 04:11:00 dist-test-slave-00ld krb5kdc[13702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223059, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 24 04:11:00 dist-test-slave-00ld krb5kdc[13702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223060, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:11:00.068890 13937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:00.055686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53880 (local address 127.0.98.60:37463)
0924 04:11:00.056093 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0924 04:11:00.056107 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 04:11:00.056140 (+ 33us) server_negotiation.cc:407] Connection header received
0924 04:11:00.056933 (+ 793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:11:00.056951 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:11:00.057274 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:11:00.057593 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:11:00.058380 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:00.059410 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:00.060267 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:00.061044 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:00.061256 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:11:00.063693 (+ 2437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:11:00.063699 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:11:00.063710 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:11:00.063743 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:11:00.065555 (+ 1812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:11:00.066038 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:11:00.066043 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:11:00.066050 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:11:00.066185 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:11:00.066608 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:11:00.066611 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:11:00.066612 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:11:00.067178 (+ 566us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:11:00.067388 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0924 04:11:00.067599 (+ 211us) server_negotiation.cc:299] Negotiation successful
0924 04:11:00.067881 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":124,"threads_started":1}
I20260924 04:11:00.069445 13938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:00.060393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39917 (local address 127.0.0.1:60174)
0924 04:11:00.060828 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0924 04:11:00.060877 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 04:11:00.060995 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:11:00.061320 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:11:00.061327 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:11:00.061781 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:11:00.061953 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:11:00.061960 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:11:00.062674 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:11:00.062677 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:11:00.063373 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:11:00.063382 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:11:00.063903 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:11:00.063906 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:11:00.063993 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:11:00.064502 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:11:00.064514 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:11:00.065926 (+ 1412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:11:00.067946 (+ 2020us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:11:00.067953 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:11:00.067956 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:11:00.068161 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:11:00.068432 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:11:00.068435 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:11:00.068436 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:11:00.068502 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:11:00.068943 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:11:00.068947 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:11:00.069098 (+ 151us) client_negotiation.cc:769] Sending connection context
0924 04:11:00.069215 (+ 117us) client_negotiation.cc:240] Negotiation successful
0924 04:11:00.069335 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":157,"threads_started":1}
I20260924 04:11:00.069542 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:00.060467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60174 (local address 127.0.98.62:39917)
0924 04:11:00.060616 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 04:11:00.060621 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:11:00.060958 (+ 337us) server_negotiation.cc:407] Connection header received
0924 04:11:00.061095 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:11:00.061098 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:11:00.061152 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:11:00.061260 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:11:00.062078 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:00.062555 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:00.063490 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:00.063786 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:00.063904 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:11:00.066079 (+ 2175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:11:00.066084 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:11:00.066090 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:11:00.066118 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:11:00.067844 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:11:00.068255 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:11:00.068259 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:11:00.068260 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:11:00.068344 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:11:00.068599 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:11:00.068602 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:11:00.068604 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:11:00.068846 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:11:00.068953 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 04:11:00.069289 (+ 336us) server_negotiation.cc:299] Negotiation successful
0924 04:11:00.069401 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 04:11:00.070420 13934 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44899 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 04:11:00 dist-test-slave-00ld krb5kdc[13702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223060, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 24 04:11:00 dist-test-slave-00ld krb5kdc[13702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223059, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:11:00.080189 13938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:00.070907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44899 (local address 127.0.0.1:58196)
0924 04:11:00.071152 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0924 04:11:00.071166 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:11:00.071250 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:11:00.071539 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:11:00.071542 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:11:00.071709 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:11:00.071840 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:11:00.071849 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:11:00.072902 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:11:00.072905 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:11:00.073735 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:11:00.073744 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:11:00.074359 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:11:00.074363 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:11:00.074450 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:11:00.074974 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:11:00.074993 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:11:00.076333 (+ 1340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:11:00.078530 (+ 2197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:11:00.078536 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:11:00.078540 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:11:00.078776 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:11:00.079078 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:11:00.079081 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:11:00.079082 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:11:00.079157 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:11:00.079680 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:11:00.079683 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:11:00.079772 (+ 89us) client_negotiation.cc:769] Sending connection context
0924 04:11:00.079883 (+ 111us) client_negotiation.cc:240] Negotiation successful
0924 04:11:00.080018 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178}
I20260924 04:11:00.080255 13863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:00.071023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58196 (local address 127.0.98.61:44899)
0924 04:11:00.071147 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 04:11:00.071152 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:11:00.071208 (+ 56us) server_negotiation.cc:407] Connection header received
0924 04:11:00.071367 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:11:00.071370 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:11:00.071419 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:11:00.071509 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:11:00.071961 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:00.072792 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:00.073850 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:00.074240 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:00.074346 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:11:00.076462 (+ 2116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:11:00.076465 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:11:00.076467 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:11:00.076488 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:11:00.078419 (+ 1931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:11:00.078875 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:11:00.078878 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:11:00.078880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:11:00.078981 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:11:00.079280 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:11:00.079283 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:11:00.079284 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:11:00.079584 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:11:00.079675 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 04:11:00.079974 (+ 299us) server_negotiation.cc:299] Negotiation successful
0924 04:11:00.080089 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 04:11:00.080459 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:00.071095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60186 (local address 127.0.98.62:39917)
0924 04:11:00.071187 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0924 04:11:00.071190 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:11:00.071248 (+ 58us) server_negotiation.cc:407] Connection header received
0924 04:11:00.071378 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:11:00.071380 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:11:00.071419 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:11:00.071501 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:11:00.072181 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:00.072893 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:00.073913 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:00.074240 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:00.074346 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:11:00.077207 (+ 2861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:11:00.077211 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:11:00.077213 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:11:00.077248 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:11:00.078736 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:11:00.079246 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:11:00.079249 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:11:00.079250 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:11:00.079336 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:11:00.079598 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:11:00.079600 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:11:00.079602 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:11:00.079803 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:11:00.080033 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0924 04:11:00.080185 (+ 152us) server_negotiation.cc:299] Negotiation successful
0924 04:11:00.080301 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 04:11:00.080852 13934 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37463 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:11:00.082613 13783 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37463 } attempt: 3
I20260924 04:11:00.085578 13858 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37463 } attempt: 3
Sep 24 04:11:00 dist-test-slave-00ld krb5kdc[13702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223060, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 24 04:11:00 dist-test-slave-00ld krb5kdc[13702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223059, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 24 04:11:00 dist-test-slave-00ld krb5kdc[13702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223059, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 04:11:00.090066 13938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:00.081315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37463 (local address 127.0.0.1:53896)
0924 04:11:00.081483 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 04:11:00.081494 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:11:00.081566 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:11:00.081993 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:11:00.081995 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:11:00.082113 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:11:00.082212 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:11:00.082218 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:11:00.083600 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:11:00.083603 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:11:00.084334 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:11:00.084341 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:11:00.084807 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:11:00.084811 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:11:00.084882 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:11:00.085332 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:11:00.085345 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:11:00.086699 (+ 1354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:11:00.088671 (+ 1972us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:11:00.088673 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:11:00.088674 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:11:00.088884 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:11:00.089163 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:11:00.089165 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:11:00.089167 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:11:00.089241 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:11:00.089602 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:11:00.089604 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:11:00.089668 (+ 64us) client_negotiation.cc:769] Sending connection context
0924 04:11:00.089748 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 04:11:00.089869 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"mutex_wait_us":21}
I20260924 04:11:00.090631 13937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:00.081386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53896 (local address 127.0.98.60:37463)
0924 04:11:00.081492 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0924 04:11:00.081498 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:11:00.081536 (+ 38us) server_negotiation.cc:407] Connection header received
0924 04:11:00.081654 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:11:00.081657 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:11:00.081702 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:11:00.081795 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:11:00.082323 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:00.083488 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:00.084436 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:00.084674 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:00.084813 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:11:00.086822 (+ 2009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:11:00.086828 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:11:00.086833 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:11:00.086861 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:11:00.088538 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:11:00.088977 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:11:00.088982 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:11:00.088983 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:11:00.089069 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:11:00.089323 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:11:00.089326 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:11:00.089327 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:11:00.089494 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:11:00.089573 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 04:11:00.090437 (+ 864us) server_negotiation.cc:299] Negotiation successful
0924 04:11:00.090543 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 04:11:00.093154 13787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:00.082966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37463 (local address 127.0.0.1:53910)
0924 04:11:00.083117 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 04:11:00.083130 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:11:00.083194 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:11:00.083577 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:11:00.083580 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:11:00.083774 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:11:00.083847 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:11:00.083853 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:11:00.084907 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:11:00.084911 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:11:00.085699 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:11:00.085712 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:11:00.086366 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:11:00.086370 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:11:00.086469 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:11:00.086886 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:11:00.086903 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:11:00.088905 (+ 2002us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:11:00.091455 (+ 2550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:11:00.091457 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:11:00.091458 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:11:00.091733 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:11:00.092050 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:11:00.092053 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:11:00.092057 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:11:00.092149 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:11:00.092577 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:11:00.092582 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:11:00.092732 (+ 150us) client_negotiation.cc:769] Sending connection context
0924 04:11:00.092853 (+ 121us) client_negotiation.cc:240] Negotiation successful
0924 04:11:00.092990 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93}
I20260924 04:11:00.093168 13939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:00.083049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53910 (local address 127.0.98.60:37463)
0924 04:11:00.083369 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0924 04:11:00.083377 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 04:11:00.083387 (+ 10us) server_negotiation.cc:407] Connection header received
0924 04:11:00.083426 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:11:00.083428 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:11:00.083457 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:11:00.083560 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:11:00.083955 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:00.084785 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:00.085840 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:00.086216 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:00.086680 (+ 464us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:11:00.089077 (+ 2397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:11:00.089081 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:11:00.089083 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:11:00.089106 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:11:00.091349 (+ 2243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:11:00.091852 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:11:00.091856 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:11:00.091857 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:11:00.091944 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:11:00.092265 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:11:00.092269 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:11:00.092271 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:11:00.092469 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:11:00.092575 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 04:11:00.092902 (+ 327us) server_negotiation.cc:299] Negotiation successful
0924 04:11:00.093005 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":260,"thread_start_us":97,"threads_started":1}
Sep 24 04:11:00 dist-test-slave-00ld krb5kdc[13702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223059, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 04:11:00.095266 13863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:00.082229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58204 (local address 127.0.98.61:44899)
0924 04:11:00.082343 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0924 04:11:00.082349 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:11:00.082362 (+ 13us) server_negotiation.cc:407] Connection header received
0924 04:11:00.082405 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:11:00.082408 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:11:00.082441 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:11:00.082523 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:11:00.083329 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:00.084066 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:00.085280 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:00.085624 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:00.085748 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:11:00.089556 (+ 3808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:11:00.089562 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:11:00.089565 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:11:00.089592 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:11:00.091480 (+ 1888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:11:00.091974 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:11:00.091980 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:11:00.091985 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:11:00.092094 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:11:00.093931 (+ 1837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:11:00.093934 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:11:00.093935 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:11:00.094147 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:11:00.094371 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0924 04:11:00.095013 (+ 642us) server_negotiation.cc:299] Negotiation successful
0924 04:11:00.095127 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 04:11:00.095830 13934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d62f8761fba24465beb67ac51480840e: Bootstrap starting.
I20260924 04:11:00.096647 13783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9485a46effd14f048c7046ae9da7bb92: Bootstrap starting.
I20260924 04:11:00.097711 13861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:00.085950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37463 (local address 127.0.0.1:53920)
0924 04:11:00.086427 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0924 04:11:00.086442 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:11:00.086506 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:11:00.087706 (+ 1200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:11:00.087709 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:11:00.087872 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:11:00.087948 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:11:00.087952 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:11:00.089034 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:11:00.089040 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:11:00.089869 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:11:00.089878 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:11:00.091897 (+ 2019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:11:00.091901 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:11:00.091983 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:11:00.092434 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:11:00.092451 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:11:00.093791 (+ 1340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:11:00.095927 (+ 2136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:11:00.095930 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:11:00.095932 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:11:00.096199 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:11:00.096612 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:11:00.096615 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:11:00.096617 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:11:00.096747 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:11:00.097227 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:11:00.097230 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:11:00.097321 (+ 91us) client_negotiation.cc:769] Sending connection context
0924 04:11:00.097429 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 04:11:00.097568 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":394}
I20260924 04:11:00.097786 13940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:00.086956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53920 (local address 127.0.98.60:37463)
0924 04:11:00.087499 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0924 04:11:00.087509 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 04:11:00.087523 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:11:00.087559 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:11:00.087565 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:11:00.087613 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:11:00.087692 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:11:00.088094 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:00.088946 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:00.090008 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:00.091728 (+ 1720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:00.092193 (+ 465us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:11:00.093916 (+ 1723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:11:00.093921 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:11:00.093926 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:11:00.093952 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:11:00.095811 (+ 1859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:11:00.096306 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:11:00.096309 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:11:00.096311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:11:00.096394 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:11:00.096858 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:11:00.096862 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:11:00.096863 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:11:00.097070 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:11:00.097202 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0924 04:11:00.097510 (+ 308us) server_negotiation.cc:299] Negotiation successful
0924 04:11:00.097630 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":368,"threads_started":1}
I20260924 04:11:00.099011 13934 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d62f8761fba24465beb67ac51480840e: Neither blocks nor log segments found. Creating new log.
I20260924 04:11:00.099257 13783 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9485a46effd14f048c7046ae9da7bb92: Neither blocks nor log segments found. Creating new log.
I20260924 04:11:00.099684 13934 log.cc:824] T 00000000000000000000000000000000 P d62f8761fba24465beb67ac51480840e: Log is configured to *not* fsync() on all Append() calls
I20260924 04:11:00.100031 13783 log.cc:824] T 00000000000000000000000000000000 P 9485a46effd14f048c7046ae9da7bb92: Log is configured to *not* fsync() on all Append() calls
I20260924 04:11:00.101044 13858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8b0bf0e2e534ce19ce3934bd892bab2: Bootstrap starting.
I20260924 04:11:00.101905 13934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d62f8761fba24465beb67ac51480840e: No bootstrap required, opened a new log
I20260924 04:11:00.103657 13858 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b8b0bf0e2e534ce19ce3934bd892bab2: Neither blocks nor log segments found. Creating new log.
I20260924 04:11:00.103969 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:11:00.104449 13858 log.cc:824] T 00000000000000000000000000000000 P b8b0bf0e2e534ce19ce3934bd892bab2: Log is configured to *not* fsync() on all Append() calls
I20260924 04:11:00.106148 13934 raft_consensus.cc:348] T 00000000000000000000000000000000 P d62f8761fba24465beb67ac51480840e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9485a46effd14f048c7046ae9da7bb92" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39917 } } peers { permanent_uuid: "b8b0bf0e2e534ce19ce3934bd892bab2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44899 } } peers { permanent_uuid: "d62f8761fba24465beb67ac51480840e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37463 } }
I20260924 04:11:00.106904 13934 raft_consensus.cc:374] T 00000000000000000000000000000000 P d62f8761fba24465beb67ac51480840e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:11:00.106958 13934 raft_consensus.cc:729] T 00000000000000000000000000000000 P d62f8761fba24465beb67ac51480840e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d62f8761fba24465beb67ac51480840e, State: Initialized, Role: FOLLOWER
I20260924 04:11:00.107509 13934 consensus_queue.cc:260] T 00000000000000000000000000000000 P d62f8761fba24465beb67ac51480840e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9485a46effd14f048c7046ae9da7bb92" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39917 } } peers { permanent_uuid: "b8b0bf0e2e534ce19ce3934bd892bab2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44899 } } peers { permanent_uuid: "d62f8761fba24465beb67ac51480840e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37463 } }
I20260924 04:11:00.107865 13858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8b0bf0e2e534ce19ce3934bd892bab2: No bootstrap required, opened a new log
I20260924 04:11:00.108042 13783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9485a46effd14f048c7046ae9da7bb92: No bootstrap required, opened a new log
I20260924 04:11:00.108870 13934 sys_catalog.cc:564] T 00000000000000000000000000000000 P d62f8761fba24465beb67ac51480840e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:11:00.112890 13783 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9485a46effd14f048c7046ae9da7bb92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9485a46effd14f048c7046ae9da7bb92" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39917 } } peers { permanent_uuid: "b8b0bf0e2e534ce19ce3934bd892bab2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44899 } } peers { permanent_uuid: "d62f8761fba24465beb67ac51480840e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37463 } }
I20260924 04:11:00.113282 13783 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9485a46effd14f048c7046ae9da7bb92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:11:00.113394 13783 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9485a46effd14f048c7046ae9da7bb92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9485a46effd14f048c7046ae9da7bb92, State: Initialized, Role: FOLLOWER
I20260924 04:11:00.113904 13783 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9485a46effd14f048c7046ae9da7bb92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9485a46effd14f048c7046ae9da7bb92" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39917 } } peers { permanent_uuid: "b8b0bf0e2e534ce19ce3934bd892bab2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44899 } } peers { permanent_uuid: "d62f8761fba24465beb67ac51480840e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37463 } }
I20260924 04:11:00.115433 13783 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9485a46effd14f048c7046ae9da7bb92 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:11:00.112351 13865 master_runner.cc:186] Error getting master registration for 127.0.98.62:39917: OK, instance_id { permanent_uuid: "9485a46effd14f048c7046ae9da7bb92" instance_seqno: 1790223059614724 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 04:11:00.115764 13865 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 04:11:00.117168 13858 raft_consensus.cc:348] T 00000000000000000000000000000000 P b8b0bf0e2e534ce19ce3934bd892bab2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9485a46effd14f048c7046ae9da7bb92" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39917 } } peers { permanent_uuid: "b8b0bf0e2e534ce19ce3934bd892bab2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44899 } } peers { permanent_uuid: "d62f8761fba24465beb67ac51480840e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37463 } }
I20260924 04:11:00.117390 13858 raft_consensus.cc:374] T 00000000000000000000000000000000 P b8b0bf0e2e534ce19ce3934bd892bab2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:11:00.117444 13858 raft_consensus.cc:729] T 00000000000000000000000000000000 P b8b0bf0e2e534ce19ce3934bd892bab2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8b0bf0e2e534ce19ce3934bd892bab2, State: Initialized, Role: FOLLOWER
I20260924 04:11:00.117890 13858 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8b0bf0e2e534ce19ce3934bd892bab2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9485a46effd14f048c7046ae9da7bb92" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39917 } } peers { permanent_uuid: "b8b0bf0e2e534ce19ce3934bd892bab2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44899 } } peers { permanent_uuid: "d62f8761fba24465beb67ac51480840e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37463 } }
I20260924 04:11:00.118366 13940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:00.106760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53932 (local address 127.0.98.60:37463)
0924 04:11:00.106970 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0924 04:11:00.106974 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:11:00.107459 (+ 485us) server_negotiation.cc:407] Connection header received
0924 04:11:00.107646 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:11:00.107649 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:11:00.107696 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:11:00.107745 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:11:00.108752 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:00.109589 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:00.110771 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:00.111144 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:00.111219 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:11:00.113717 (+ 2498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:11:00.113721 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:11:00.113724 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:11:00.113749 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:11:00.115537 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:11:00.116044 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:11:00.116048 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:11:00.116050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:11:00.116405 (+ 355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:11:00.116801 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:11:00.116805 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:11:00.116807 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:11:00.117024 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:11:00.117156 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0924 04:11:00.117819 (+ 663us) server_negotiation.cc:299] Negotiation successful
0924 04:11:00.117954 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150}
I20260924 04:11:00.119236 13858 sys_catalog.cc:564] T 00000000000000000000000000000000 P b8b0bf0e2e534ce19ce3934bd892bab2 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:11:00.121626 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:00.106638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60194 (local address 127.0.98.62:39917)
0924 04:11:00.106868 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0924 04:11:00.106873 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:11:00.111884 (+ 5011us) server_negotiation.cc:407] Connection header received
0924 04:11:00.112337 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:11:00.112343 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:11:00.112393 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:11:00.112446 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:11:00.113357 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:00.114087 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:00.115242 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:00.115579 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:00.115659 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:11:00.117320 (+ 1661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:11:00.117324 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:11:00.117327 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:11:00.117353 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:11:00.118926 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:11:00.119738 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:11:00.119742 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:11:00.119744 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:11:00.119843 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:11:00.120185 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:11:00.120189 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:11:00.120191 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:11:00.120384 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:11:00.120520 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 04:11:00.121305 (+ 785us) server_negotiation.cc:299] Negotiation successful
0924 04:11:00.121435 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161}
I20260924 04:11:00.122102 13974 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8b0bf0e2e534ce19ce3934bd892bab2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9485a46effd14f048c7046ae9da7bb92" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39917 } } peers { permanent_uuid: "b8b0bf0e2e534ce19ce3934bd892bab2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44899 } } peers { permanent_uuid: "d62f8761fba24465beb67ac51480840e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37463 } } }
I20260924 04:11:00.122218 13974 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8b0bf0e2e534ce19ce3934bd892bab2 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:11:00.122771 13951 sys_catalog.cc:455] T 00000000000000000000000000000000 P d62f8761fba24465beb67ac51480840e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9485a46effd14f048c7046ae9da7bb92" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39917 } } peers { permanent_uuid: "b8b0bf0e2e534ce19ce3934bd892bab2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44899 } } peers { permanent_uuid: "d62f8761fba24465beb67ac51480840e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37463 } } }
I20260924 04:11:00.122913 13951 sys_catalog.cc:458] T 00000000000000000000000000000000 P d62f8761fba24465beb67ac51480840e [sys.catalog]: This master's current role is: FOLLOWER
W20260924 04:11:00.124665 13962 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d62f8761fba24465beb67ac51480840e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:11:00.124778 13962 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260924 04:11:00.127315 13986 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b8b0bf0e2e534ce19ce3934bd892bab2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:11:00.127405 13986 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:11:00.128315 13715 master_runner.cc:263] No leader master found from master 9485a46effd14f048c7046ae9da7bb92
I20260924 04:11:00.128376 13715 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 04:11:00.128424 13790 master_runner.cc:263] No leader master found from master b8b0bf0e2e534ce19ce3934bd892bab2
I20260924 04:11:00.128479 13790 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 04:11:00.129220 13863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:00.106278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58216 (local address 127.0.98.61:44899)
0924 04:11:00.106402 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 04:11:00.106408 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:11:00.111086 (+ 4678us) server_negotiation.cc:407] Connection header received
0924 04:11:00.111267 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:11:00.111271 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:11:00.111313 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:11:00.111359 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:11:00.119034 (+ 7675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:00.119712 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:00.123103 (+ 3391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:00.123412 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:00.123511 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:11:00.125409 (+ 1898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:11:00.125413 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:11:00.125415 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:11:00.125438 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:11:00.127110 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:11:00.127720 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:11:00.127724 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:11:00.127725 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:11:00.127806 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:11:00.128126 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:11:00.128130 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:11:00.128132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:11:00.128357 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:11:00.128468 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 04:11:00.128933 (+ 465us) server_negotiation.cc:299] Negotiation successful
0924 04:11:00.129049 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260924 04:11:00.130760 13975 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9485a46effd14f048c7046ae9da7bb92: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:11:00.130836 13975 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:11:00.132829 13963 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9485a46effd14f048c7046ae9da7bb92 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9485a46effd14f048c7046ae9da7bb92" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39917 } } peers { permanent_uuid: "b8b0bf0e2e534ce19ce3934bd892bab2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44899 } } peers { permanent_uuid: "d62f8761fba24465beb67ac51480840e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37463 } } }
I20260924 04:11:00.132953 13963 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9485a46effd14f048c7046ae9da7bb92 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:11:00.182961 13974 raft_consensus.cc:482] T 00000000000000000000000000000000 P b8b0bf0e2e534ce19ce3934bd892bab2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:11:00.183162 13974 raft_consensus.cc:504] T 00000000000000000000000000000000 P b8b0bf0e2e534ce19ce3934bd892bab2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9485a46effd14f048c7046ae9da7bb92" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39917 } } peers { permanent_uuid: "b8b0bf0e2e534ce19ce3934bd892bab2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44899 } } peers { permanent_uuid: "d62f8761fba24465beb67ac51480840e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37463 } }
I20260924 04:11:00.184232 13974 leader_election.cc:288] T 00000000000000000000000000000000 P b8b0bf0e2e534ce19ce3934bd892bab2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9485a46effd14f048c7046ae9da7bb92 (127.0.98.62:39917), d62f8761fba24465beb67ac51480840e (127.0.98.60:37463)
I20260924 04:11:00.184904 13907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b8b0bf0e2e534ce19ce3934bd892bab2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d62f8761fba24465beb67ac51480840e" is_pre_election: true
I20260924 04:11:00.185237 13907 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d62f8761fba24465beb67ac51480840e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8b0bf0e2e534ce19ce3934bd892bab2 in term 0.
I20260924 04:11:00.185622 13808 leader_election.cc:302] T 00000000000000000000000000000000 P b8b0bf0e2e534ce19ce3934bd892bab2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b8b0bf0e2e534ce19ce3934bd892bab2, d62f8761fba24465beb67ac51480840e; no voters:
I20260924 04:11:00.186394 13974 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b8b0bf0e2e534ce19ce3934bd892bab2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:11:00.186336 13757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b8b0bf0e2e534ce19ce3934bd892bab2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9485a46effd14f048c7046ae9da7bb92" is_pre_election: true
I20260924 04:11:00.186491 13974 raft_consensus.cc:482] T 00000000000000000000000000000000 P b8b0bf0e2e534ce19ce3934bd892bab2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:11:00.186535 13974 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b8b0bf0e2e534ce19ce3934bd892bab2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:11:00.186756 13757 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9485a46effd14f048c7046ae9da7bb92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8b0bf0e2e534ce19ce3934bd892bab2 in term 0.
I20260924 04:11:00.187844 13974 raft_consensus.cc:504] T 00000000000000000000000000000000 P b8b0bf0e2e534ce19ce3934bd892bab2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9485a46effd14f048c7046ae9da7bb92" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39917 } } peers { permanent_uuid: "b8b0bf0e2e534ce19ce3934bd892bab2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44899 } } peers { permanent_uuid: "d62f8761fba24465beb67ac51480840e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37463 } }
I20260924 04:11:00.188175 13974 leader_election.cc:288] T 00000000000000000000000000000000 P b8b0bf0e2e534ce19ce3934bd892bab2 [CANDIDATE]: Term 1 election: Requested vote from peers 9485a46effd14f048c7046ae9da7bb92 (127.0.98.62:39917), d62f8761fba24465beb67ac51480840e (127.0.98.60:37463)
I20260924 04:11:00.188946 13757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b8b0bf0e2e534ce19ce3934bd892bab2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9485a46effd14f048c7046ae9da7bb92"
I20260924 04:11:00.189075 13757 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9485a46effd14f048c7046ae9da7bb92 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:11:00.190347 13757 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9485a46effd14f048c7046ae9da7bb92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8b0bf0e2e534ce19ce3934bd892bab2 in term 1.
I20260924 04:11:00.190726 13809 leader_election.cc:302] T 00000000000000000000000000000000 P b8b0bf0e2e534ce19ce3934bd892bab2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9485a46effd14f048c7046ae9da7bb92, b8b0bf0e2e534ce19ce3934bd892bab2; no voters:
I20260924 04:11:00.190970 13907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b8b0bf0e2e534ce19ce3934bd892bab2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d62f8761fba24465beb67ac51480840e"
I20260924 04:11:00.191104 13907 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d62f8761fba24465beb67ac51480840e [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:11:00.192371 13907 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d62f8761fba24465beb67ac51480840e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8b0bf0e2e534ce19ce3934bd892bab2 in term 1.
I20260924 04:11:00.195196 13974 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b8b0bf0e2e534ce19ce3934bd892bab2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:11:00.195492 13974 raft_consensus.cc:686] T 00000000000000000000000000000000 P b8b0bf0e2e534ce19ce3934bd892bab2 [term 1 LEADER]: Becoming Leader. State: Replica: b8b0bf0e2e534ce19ce3934bd892bab2, State: Running, Role: LEADER
I20260924 04:11:00.196053 13974 consensus_queue.cc:237] T 00000000000000000000000000000000 P b8b0bf0e2e534ce19ce3934bd892bab2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9485a46effd14f048c7046ae9da7bb92" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39917 } } peers { permanent_uuid: "b8b0bf0e2e534ce19ce3934bd892bab2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44899 } } peers { permanent_uuid: "d62f8761fba24465beb67ac51480840e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37463 } }
I20260924 04:11:00.198611 13974 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8b0bf0e2e534ce19ce3934bd892bab2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8b0bf0e2e534ce19ce3934bd892bab2. Latest consensus state: current_term: 1 leader_uuid: "b8b0bf0e2e534ce19ce3934bd892bab2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9485a46effd14f048c7046ae9da7bb92" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39917 } } peers { permanent_uuid: "b8b0bf0e2e534ce19ce3934bd892bab2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44899 } } peers { permanent_uuid: "d62f8761fba24465beb67ac51480840e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37463 } } }
I20260924 04:11:00.198786 13974 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8b0bf0e2e534ce19ce3934bd892bab2 [sys.catalog]: This master's current role is: LEADER
I20260924 04:11:00.199220 13992 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:11:00.200204 13992 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:11:00.207600 13907 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d62f8761fba24465beb67ac51480840e [term 1 FOLLOWER]: Refusing update from remote peer b8b0bf0e2e534ce19ce3934bd892bab2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:11:00.210754 13990 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b8b0bf0e2e534ce19ce3934bd892bab2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d62f8761fba24465beb67ac51480840e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:11:00.215075 13951 sys_catalog.cc:455] T 00000000000000000000000000000000 P d62f8761fba24465beb67ac51480840e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8b0bf0e2e534ce19ce3934bd892bab2. Latest consensus state: current_term: 1 leader_uuid: "b8b0bf0e2e534ce19ce3934bd892bab2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9485a46effd14f048c7046ae9da7bb92" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39917 } } peers { permanent_uuid: "b8b0bf0e2e534ce19ce3934bd892bab2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44899 } } peers { permanent_uuid: "d62f8761fba24465beb67ac51480840e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37463 } } }
I20260924 04:11:00.215193 13951 sys_catalog.cc:458] T 00000000000000000000000000000000 P d62f8761fba24465beb67ac51480840e [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:11:00.216231 13757 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9485a46effd14f048c7046ae9da7bb92 [term 1 FOLLOWER]: Refusing update from remote peer b8b0bf0e2e534ce19ce3934bd892bab2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:11:00.216843 13990 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b8b0bf0e2e534ce19ce3934bd892bab2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485a46effd14f048c7046ae9da7bb92" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:11:00.220554 13963 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9485a46effd14f048c7046ae9da7bb92 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8b0bf0e2e534ce19ce3934bd892bab2. Latest consensus state: current_term: 1 leader_uuid: "b8b0bf0e2e534ce19ce3934bd892bab2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9485a46effd14f048c7046ae9da7bb92" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39917 } } peers { permanent_uuid: "b8b0bf0e2e534ce19ce3934bd892bab2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44899 } } peers { permanent_uuid: "d62f8761fba24465beb67ac51480840e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37463 } } }
I20260924 04:11:00.220700 13963 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9485a46effd14f048c7046ae9da7bb92 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:11:00.227758 13990 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8b0bf0e2e534ce19ce3934bd892bab2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b8b0bf0e2e534ce19ce3934bd892bab2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9485a46effd14f048c7046ae9da7bb92" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39917 } } peers { permanent_uuid: "b8b0bf0e2e534ce19ce3934bd892bab2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44899 } } peers { permanent_uuid: "d62f8761fba24465beb67ac51480840e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37463 } } }
I20260924 04:11:00.227885 13990 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8b0bf0e2e534ce19ce3934bd892bab2 [sys.catalog]: This master's current role is: LEADER
I20260924 04:11:00.229128 13963 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9485a46effd14f048c7046ae9da7bb92 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b8b0bf0e2e534ce19ce3934bd892bab2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9485a46effd14f048c7046ae9da7bb92" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39917 } } peers { permanent_uuid: "b8b0bf0e2e534ce19ce3934bd892bab2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44899 } } peers { permanent_uuid: "d62f8761fba24465beb67ac51480840e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37463 } } }
I20260924 04:11:00.229238 13963 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9485a46effd14f048c7046ae9da7bb92 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:11:00.229653 13993 mvcc.cc:205] Tried to move back new op lower bound from 7332753654601584640 to 7332753654566547456. Current Snapshot: MvccSnapshot[applied={T|T < 7332753654601584640}]
I20260924 04:11:00.231355 13996 mvcc.cc:205] Tried to move back new op lower bound from 7332753654601584640 to 7332753654566547456. Current Snapshot: MvccSnapshot[applied={T|T < 7332753654601584640}]
I20260924 04:11:00.232111 13974 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8b0bf0e2e534ce19ce3934bd892bab2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b8b0bf0e2e534ce19ce3934bd892bab2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9485a46effd14f048c7046ae9da7bb92" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39917 } } peers { permanent_uuid: "b8b0bf0e2e534ce19ce3934bd892bab2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44899 } } peers { permanent_uuid: "d62f8761fba24465beb67ac51480840e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37463 } } }
I20260924 04:11:00.232216 13974 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8b0bf0e2e534ce19ce3934bd892bab2 [sys.catalog]: This master's current role is: LEADER
I20260924 04:11:00.233187 13951 sys_catalog.cc:455] T 00000000000000000000000000000000 P d62f8761fba24465beb67ac51480840e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b8b0bf0e2e534ce19ce3934bd892bab2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9485a46effd14f048c7046ae9da7bb92" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39917 } } peers { permanent_uuid: "b8b0bf0e2e534ce19ce3934bd892bab2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44899 } } peers { permanent_uuid: "d62f8761fba24465beb67ac51480840e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37463 } } }
I20260924 04:11:00.233302 13951 sys_catalog.cc:458] T 00000000000000000000000000000000 P d62f8761fba24465beb67ac51480840e [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:11:00.235757 13994 mvcc.cc:205] Tried to move back new op lower bound from 7332753654601584640 to 7332753654566547456. Current Snapshot: MvccSnapshot[applied={T|T < 7332753654601584640}]
I20260924 04:11:00.242194 13992 catalog_manager.cc:1347] Generated new cluster ID: b0306c18a5854cbe8fbbe43c84e60be9
I20260924 04:11:00.242262 13992 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:11:00.279667 13992 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:11:00.281112 13992 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:11:00.301534 13992 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b8b0bf0e2e534ce19ce3934bd892bab2: Generated new TSK 0
I20260924 04:11:00.302605 13992 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:11:01.126039 13962 catalog_manager.cc:1259] Loaded cluster ID: b0306c18a5854cbe8fbbe43c84e60be9
I20260924 04:11:01.126163 13962 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d62f8761fba24465beb67ac51480840e: loading cluster ID for follower catalog manager: success
I20260924 04:11:01.127847 13962 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d62f8761fba24465beb67ac51480840e: acquiring CA information for follower catalog manager: success
I20260924 04:11:01.129052 13962 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d62f8761fba24465beb67ac51480840e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 04:11:01.131901 13975 catalog_manager.cc:1259] Loaded cluster ID: b0306c18a5854cbe8fbbe43c84e60be9
I20260924 04:11:01.131958 13975 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9485a46effd14f048c7046ae9da7bb92: loading cluster ID for follower catalog manager: success
I20260924 04:11:01.133471 13975 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9485a46effd14f048c7046ae9da7bb92: acquiring CA information for follower catalog manager: success
I20260924 04:11:01.134485 13975 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9485a46effd14f048c7046ae9da7bb92: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 04:11:01.368610 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 13715
I20260924 04:11:01.375186 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 13790
I20260924 04:11:01.380817 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 13865
2026-09-24T04:11:01Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3141 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790222898807720-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:11:01 dist-test-slave-00ld krb5kdc[14006](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:11:01 dist-test-slave-00ld krb5kdc[14006](info): set up 2 sockets
Sep 24 04:11:01 dist-test-slave-00ld krb5kdc[14006](info): commencing operation
krb5kdc: starting...
W20260924 04:11:02.425115 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790222898807720-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790222898807720-392-0/krb5kdc/test-user.keytab.
Sep 24 04:11:02 dist-test-slave-00ld krb5kdc[14006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223062, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:11:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:11:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:11:02.543581 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790222898807720-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790222898807720-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790222898807720-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37307
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37307
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:11:02.650462 14019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:11:02.650771 14019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:11:02.650851 14019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:11:02.655011 14019 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:11:02.655092 14019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:11:02.655131 14019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:11:02.655189 14019 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:11:02.655230 14019 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:11:02.660463 14019 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790222898807720-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790222898807720-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37307
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14019
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790222898807720-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:11:02.661628 14019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:11:02.662731 14019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:11:02.671674 14027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:11:02.671648 14024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:11:02.671670 14025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:11:02.671885 14019 server_base.cc:1034] running on GCE node
I20260924 04:11:02.672461 14019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:11:02.673631 14019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:11:02.674813 14019 hybrid_clock.cc:648] HybridClock initialized: now 1790223062674793 us; error 30 us; skew 500 ppm
Sep 24 04:11:02 dist-test-slave-00ld krb5kdc[14006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223062, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:11:02.677760 14019 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:11:02.678669 14019 webserver.cc:460] Webserver started at http://127.0.98.62:37077/ using document root <none> and password file <none>
I20260924 04:11:02.679246 14019 fs_manager.cc:359] Metadata directory not provided
I20260924 04:11:02.679397 14019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:11:02.679627 14019 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:11:02.681746 14019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790222898807720-392-0/minicluster-data/master-0/data/instance:
uuid: "2f1db0bdac8f48a58570f24a3c2b3ab9"
format_stamp: "Formatted at 2026-09-24 04:11:02 on dist-test-slave-00ld"
server_key: "8d178957bf4eee64037610d6b34bcf0d"
server_key_iv: "3fea127da90ee91736e8d686fd528bfe"
server_key_version: "encryptionkey@0"
I20260924 04:11:02.682331 14019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790222898807720-392-0/minicluster-data/master-0/wal/instance:
uuid: "2f1db0bdac8f48a58570f24a3c2b3ab9"
format_stamp: "Formatted at 2026-09-24 04:11:02 on dist-test-slave-00ld"
server_key: "8d178957bf4eee64037610d6b34bcf0d"
server_key_iv: "3fea127da90ee91736e8d686fd528bfe"
server_key_version: "encryptionkey@0"
I20260924 04:11:02.686023 14019 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:11:02.688525 14034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:11:02.689905 14019 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 04:11:02.690060 14019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790222898807720-392-0/minicluster-data/master-0/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790222898807720-392-0/minicluster-data/master-0/wal
uuid: "2f1db0bdac8f48a58570f24a3c2b3ab9"
format_stamp: "Formatted at 2026-09-24 04:11:02 on dist-test-slave-00ld"
server_key: "8d178957bf4eee64037610d6b34bcf0d"
server_key_iv: "3fea127da90ee91736e8d686fd528bfe"
server_key_version: "encryptionkey@0"
I20260924 04:11:02.690182 14019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790222898807720-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790222898807720-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790222898807720-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:11:02.700026 14019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:11:02.702983 14019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:11:02.703269 14019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:11:02.710319 14019 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37307
I20260924 04:11:02.710338 14086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37307 every 8 connection(s)
I20260924 04:11:02.711272 14019 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:11:02.714076 14087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:11:02.719385 14087 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f1db0bdac8f48a58570f24a3c2b3ab9: Bootstrap starting.
I20260924 04:11:02.719660 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 14019
I20260924 04:11:02.719771 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790222898807720-392-0/minicluster-data/master-0/wal/instance
I20260924 04:11:02.720005 392 external_mini_cluster.cc:1428] Setting key a73da37d9564c44e295c3afc9961e527
I20260924 04:11:02.721822 14087 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f1db0bdac8f48a58570f24a3c2b3ab9: Neither blocks nor log segments found. Creating new log.
I20260924 04:11:02.722529 14087 log.cc:824] T 00000000000000000000000000000000 P 2f1db0bdac8f48a58570f24a3c2b3ab9: Log is configured to *not* fsync() on all Append() calls
I20260924 04:11:02.724571 14087 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f1db0bdac8f48a58570f24a3c2b3ab9: No bootstrap required, opened a new log
I20260924 04:11:02.728937 14087 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f1db0bdac8f48a58570f24a3c2b3ab9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f1db0bdac8f48a58570f24a3c2b3ab9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37307 } }
I20260924 04:11:02.729147 14087 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2f1db0bdac8f48a58570f24a3c2b3ab9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
Sep 24 04:11:02 dist-test-slave-00ld krb5kdc[14006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223062, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:11:02.729204 14087 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f1db0bdac8f48a58570f24a3c2b3ab9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f1db0bdac8f48a58570f24a3c2b3ab9, State: Initialized, Role: FOLLOWER
I20260924 04:11:02.729662 14087 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f1db0bdac8f48a58570f24a3c2b3ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f1db0bdac8f48a58570f24a3c2b3ab9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37307 } }
I20260924 04:11:02.729831 14087 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f1db0bdac8f48a58570f24a3c2b3ab9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:11:02.729895 14087 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f1db0bdac8f48a58570f24a3c2b3ab9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:11:02.730000 14087 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f1db0bdac8f48a58570f24a3c2b3ab9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:11:02.731202 14087 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f1db0bdac8f48a58570f24a3c2b3ab9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f1db0bdac8f48a58570f24a3c2b3ab9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37307 } }
I20260924 04:11:02.731567 14087 leader_election.cc:302] T 00000000000000000000000000000000 P 2f1db0bdac8f48a58570f24a3c2b3ab9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f1db0bdac8f48a58570f24a3c2b3ab9; no voters:
I20260924 04:11:02.731897 14087 leader_election.cc:288] T 00000000000000000000000000000000 P 2f1db0bdac8f48a58570f24a3c2b3ab9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:11:02.732025 14092 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f1db0bdac8f48a58570f24a3c2b3ab9 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:11:02.732262 14092 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f1db0bdac8f48a58570f24a3c2b3ab9 [term 1 LEADER]: Becoming Leader. State: Replica: 2f1db0bdac8f48a58570f24a3c2b3ab9, State: Running, Role: LEADER
I20260924 04:11:02.732733 14092 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f1db0bdac8f48a58570f24a3c2b3ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f1db0bdac8f48a58570f24a3c2b3ab9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37307 } }
I20260924 04:11:02.733258 14087 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f1db0bdac8f48a58570f24a3c2b3ab9 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:11:02.733958 14094 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f1db0bdac8f48a58570f24a3c2b3ab9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f1db0bdac8f48a58570f24a3c2b3ab9. Latest consensus state: current_term: 1 leader_uuid: "2f1db0bdac8f48a58570f24a3c2b3ab9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f1db0bdac8f48a58570f24a3c2b3ab9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37307 } } }
I20260924 04:11:02.734756 14094 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f1db0bdac8f48a58570f24a3c2b3ab9 [sys.catalog]: This master's current role is: LEADER
I20260924 04:11:02.734650 14093 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f1db0bdac8f48a58570f24a3c2b3ab9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2f1db0bdac8f48a58570f24a3c2b3ab9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f1db0bdac8f48a58570f24a3c2b3ab9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37307 } } }
I20260924 04:11:02.735849 14093 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f1db0bdac8f48a58570f24a3c2b3ab9 [sys.catalog]: This master's current role is: LEADER
I20260924 04:11:02.735632 14090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:02.721246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36816 (local address 127.0.98.62:37307)
0924 04:11:02.721741 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0924 04:11:02.721757 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 04:11:02.721791 (+ 34us) server_negotiation.cc:407] Connection header received
0924 04:11:02.722448 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:11:02.722464 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:11:02.722859 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:11:02.723229 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:11:02.724048 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:02.725136 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:02.726295 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:02.726745 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:02.726875 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:11:02.729780 (+ 2905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:11:02.729789 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:11:02.729806 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:11:02.729839 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:11:02.731846 (+ 2007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:11:02.732434 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:11:02.732440 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:11:02.732447 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:11:02.732582 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:11:02.732936 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:11:02.732940 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:11:02.732941 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:11:02.733448 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:11:02.733661 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0924 04:11:02.734071 (+ 410us) server_negotiation.cc:299] Negotiation successful
0924 04:11:02.734399 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":184,"threads_started":1}
W20260924 04:11:02.738900 14106 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2f1db0bdac8f48a58570f24a3c2b3ab9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260924 04:11:02.739044 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260924 04:11:02.738972 14106 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:11:02.739853 14107 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:11:02.740806 14107 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:11:02.746490 14107 catalog_manager.cc:1347] Generated new cluster ID: f6190176e1ce4c70ad2f1cd1125243dd
I20260924 04:11:02.746556 14107 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:11:02.756968 14107 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:11:02.758220 14107 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:11:02.771767 14107 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2f1db0bdac8f48a58570f24a3c2b3ab9: Generated new TSK 0
I20260924 04:11:02.772648 14107 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:11:02.795154 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 14019
2026-09-24T04:11:02Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1412 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:11:02 dist-test-slave-00ld krb5kdc[14114](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:11:02 dist-test-slave-00ld krb5kdc[14114](info): set up 2 sockets
Sep 24 04:11:02 dist-test-slave-00ld krb5kdc[14114](info): commencing operation
krb5kdc: starting...
W20260924 04:11:03.832950 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/krb5kdc/test-user.keytab.
Sep 24 04:11:03 dist-test-slave-00ld krb5kdc[14114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223063, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:11:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:11:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:11:03.951794 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36317
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36317
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:11:04.060362 14127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:11:04.060740 14127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:11:04.060827 14127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:11:04.064955 14127 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:11:04.065050 14127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:11:04.065090 14127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:11:04.065128 14127 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:11:04.065163 14127 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:11:04.070640 14127 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36317
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14127
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:11:04.071751 14127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:11:04.072817 14127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:11:04.081303 14135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:11:04.081439 14132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:11:04.081379 14127 server_base.cc:1034] running on GCE node
W20260924 04:11:04.081319 14133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:11:04.082180 14127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:11:04.083283 14127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:11:04.084470 14127 hybrid_clock.cc:648] HybridClock initialized: now 1790223064084453 us; error 30 us; skew 500 ppm
Sep 24 04:11:04 dist-test-slave-00ld krb5kdc[14114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223064, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:11:04.087381 14127 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:11:04.088212 14127 webserver.cc:460] Webserver started at http://127.0.98.62:35995/ using document root <none> and password file <none>
I20260924 04:11:04.088793 14127 fs_manager.cc:359] Metadata directory not provided
I20260924 04:11:04.088860 14127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:11:04.089030 14127 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:11:04.091044 14127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/master-0/data/instance:
uuid: "001f36501e0c44f090cf0e3ba3a05727"
format_stamp: "Formatted at 2026-09-24 04:11:04 on dist-test-slave-00ld"
server_key: "ad4519d5f762093b834ef33571f1a933"
server_key_iv: "7c2fc7657d5745075655eafc99715370"
server_key_version: "encryptionkey@0"
I20260924 04:11:04.091676 14127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/master-0/wal/instance:
uuid: "001f36501e0c44f090cf0e3ba3a05727"
format_stamp: "Formatted at 2026-09-24 04:11:04 on dist-test-slave-00ld"
server_key: "ad4519d5f762093b834ef33571f1a933"
server_key_iv: "7c2fc7657d5745075655eafc99715370"
server_key_version: "encryptionkey@0"
I20260924 04:11:04.095463 14127 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 04:11:04.097951 14142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:11:04.099234 14127 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:11:04.099395 14127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/master-0/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/master-0/wal
uuid: "001f36501e0c44f090cf0e3ba3a05727"
format_stamp: "Formatted at 2026-09-24 04:11:04 on dist-test-slave-00ld"
server_key: "ad4519d5f762093b834ef33571f1a933"
server_key_iv: "7c2fc7657d5745075655eafc99715370"
server_key_version: "encryptionkey@0"
I20260924 04:11:04.099510 14127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:11:04.119629 14127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:11:04.122635 14127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:11:04.122841 14127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:11:04.129990 14127 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36317
I20260924 04:11:04.130275 14194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36317 every 8 connection(s)
I20260924 04:11:04.130957 14127 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:11:04.133674 14195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:11:04.137956 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 14127
I20260924 04:11:04.138116 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/master-0/wal/instance
I20260924 04:11:04.138410 392 external_mini_cluster.cc:1428] Setting key 876f33ffdd482311a964d91f5bdb8319
I20260924 04:11:04.139398 14195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 001f36501e0c44f090cf0e3ba3a05727: Bootstrap starting.
I20260924 04:11:04.141778 14195 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 001f36501e0c44f090cf0e3ba3a05727: Neither blocks nor log segments found. Creating new log.
I20260924 04:11:04.142747 14195 log.cc:824] T 00000000000000000000000000000000 P 001f36501e0c44f090cf0e3ba3a05727: Log is configured to *not* fsync() on all Append() calls
I20260924 04:11:04.144869 14195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 001f36501e0c44f090cf0e3ba3a05727: No bootstrap required, opened a new log
Sep 24 04:11:04 dist-test-slave-00ld krb5kdc[14114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223063, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:11:04.149559 14195 raft_consensus.cc:348] T 00000000000000000000000000000000 P 001f36501e0c44f090cf0e3ba3a05727 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "001f36501e0c44f090cf0e3ba3a05727" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36317 } }
I20260924 04:11:04.149884 14195 raft_consensus.cc:374] T 00000000000000000000000000000000 P 001f36501e0c44f090cf0e3ba3a05727 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:11:04.149935 14195 raft_consensus.cc:729] T 00000000000000000000000000000000 P 001f36501e0c44f090cf0e3ba3a05727 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 001f36501e0c44f090cf0e3ba3a05727, State: Initialized, Role: FOLLOWER
I20260924 04:11:04.150475 14195 consensus_queue.cc:260] T 00000000000000000000000000000000 P 001f36501e0c44f090cf0e3ba3a05727 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "001f36501e0c44f090cf0e3ba3a05727" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36317 } }
I20260924 04:11:04.150663 14195 raft_consensus.cc:388] T 00000000000000000000000000000000 P 001f36501e0c44f090cf0e3ba3a05727 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:11:04.150764 14195 raft_consensus.cc:482] T 00000000000000000000000000000000 P 001f36501e0c44f090cf0e3ba3a05727 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:11:04.150888 14195 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 001f36501e0c44f090cf0e3ba3a05727 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:11:04.152086 14195 raft_consensus.cc:504] T 00000000000000000000000000000000 P 001f36501e0c44f090cf0e3ba3a05727 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "001f36501e0c44f090cf0e3ba3a05727" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36317 } }
I20260924 04:11:04.152464 14195 leader_election.cc:302] T 00000000000000000000000000000000 P 001f36501e0c44f090cf0e3ba3a05727 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 001f36501e0c44f090cf0e3ba3a05727; no voters:
I20260924 04:11:04.152863 14195 leader_election.cc:288] T 00000000000000000000000000000000 P 001f36501e0c44f090cf0e3ba3a05727 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:11:04.153141 14200 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 001f36501e0c44f090cf0e3ba3a05727 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:11:04.153424 14200 raft_consensus.cc:686] T 00000000000000000000000000000000 P 001f36501e0c44f090cf0e3ba3a05727 [term 1 LEADER]: Becoming Leader. State: Replica: 001f36501e0c44f090cf0e3ba3a05727, State: Running, Role: LEADER
I20260924 04:11:04.153756 14200 consensus_queue.cc:237] T 00000000000000000000000000000000 P 001f36501e0c44f090cf0e3ba3a05727 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "001f36501e0c44f090cf0e3ba3a05727" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36317 } }
I20260924 04:11:04.154043 14198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:04.139793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56316 (local address 127.0.98.62:36317)
0924 04:11:04.140212 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0924 04:11:04.140228 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 04:11:04.140263 (+ 35us) server_negotiation.cc:407] Connection header received
0924 04:11:04.140988 (+ 725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:11:04.141008 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:11:04.141340 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:11:04.141623 (+ 283us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:11:04.142630 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:04.143643 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:04.144791 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:04.145315 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:04.145450 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:11:04.148232 (+ 2782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:11:04.148239 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:11:04.148257 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:11:04.148291 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:11:04.150541 (+ 2250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:11:04.151149 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:11:04.151156 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:11:04.151161 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:11:04.151300 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:11:04.151617 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:11:04.151620 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:11:04.151621 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:11:04.152203 (+ 582us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:11:04.152387 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0924 04:11:04.152720 (+ 333us) server_negotiation.cc:299] Negotiation successful
0924 04:11:04.153051 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":138,"threads_started":1}
I20260924 04:11:04.154155 14195 sys_catalog.cc:564] T 00000000000000000000000000000000 P 001f36501e0c44f090cf0e3ba3a05727 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:11:04.155097 14200 sys_catalog.cc:455] T 00000000000000000000000000000000 P 001f36501e0c44f090cf0e3ba3a05727 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 001f36501e0c44f090cf0e3ba3a05727. Latest consensus state: current_term: 1 leader_uuid: "001f36501e0c44f090cf0e3ba3a05727" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "001f36501e0c44f090cf0e3ba3a05727" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36317 } } }
I20260924 04:11:04.155220 14200 sys_catalog.cc:458] T 00000000000000000000000000000000 P 001f36501e0c44f090cf0e3ba3a05727 [sys.catalog]: This master's current role is: LEADER
I20260924 04:11:04.155555 14201 sys_catalog.cc:455] T 00000000000000000000000000000000 P 001f36501e0c44f090cf0e3ba3a05727 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "001f36501e0c44f090cf0e3ba3a05727" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "001f36501e0c44f090cf0e3ba3a05727" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36317 } } }
I20260924 04:11:04.155642 14201 sys_catalog.cc:458] T 00000000000000000000000000000000 P 001f36501e0c44f090cf0e3ba3a05727 [sys.catalog]: This master's current role is: LEADER
W20260924 04:11:04.159224 14215 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 001f36501e0c44f090cf0e3ba3a05727: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:11:04.159317 14215 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:11:04.159382 14209 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:11:04.160218 14209 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:11:04.166011 14209 catalog_manager.cc:1347] Generated new cluster ID: 34b384b4f485405d98a779ee1e519a62
I20260924 04:11:04.166105 14209 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:11:04.173295 14209 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:11:04.174249 14209 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:11:04.179715 14209 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 001f36501e0c44f090cf0e3ba3a05727: Generated new TSK 0
I20260924 04:11:04.180359 14209 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:11:04.209743 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36317
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:11:04.333702 14221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:11:04.333976 14221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:11:04.334021 14221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:11:04.338112 14221 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:11:04.338172 14221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:11:04.338261 14221 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:11:04.344030 14221 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36317
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14221
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:11:04.345515 14221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:11:04.346732 14221 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:11:04.356251 14229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:11:04.356240 14227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:11:04.356560 14221 server_base.cc:1034] running on GCE node
W20260924 04:11:04.356251 14226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:11:04.357030 14221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:11:04.357698 14221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:11:04.358911 14221 hybrid_clock.cc:648] HybridClock initialized: now 1790223064358898 us; error 31 us; skew 500 ppm
Sep 24 04:11:04 dist-test-slave-00ld krb5kdc[14114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223064, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:11:04.361780 14221 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:11:04.362708 14221 webserver.cc:460] Webserver started at http://127.0.98.1:34783/ using document root <none> and password file <none>
I20260924 04:11:04.363297 14221 fs_manager.cc:359] Metadata directory not provided
I20260924 04:11:04.363390 14221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:11:04.363605 14221 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:11:04.365770 14221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-0/data/instance:
uuid: "a808f529633040fa88cd3449dd7017dc"
format_stamp: "Formatted at 2026-09-24 04:11:04 on dist-test-slave-00ld"
server_key: "c802f9bf344a5ec4b2682c6437d62edd"
server_key_iv: "1e3005a93af7b3af26bb524be11f2c12"
server_key_version: "encryptionkey@0"
I20260924 04:11:04.366334 14221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a808f529633040fa88cd3449dd7017dc"
format_stamp: "Formatted at 2026-09-24 04:11:04 on dist-test-slave-00ld"
server_key: "c802f9bf344a5ec4b2682c6437d62edd"
server_key_iv: "1e3005a93af7b3af26bb524be11f2c12"
server_key_version: "encryptionkey@0"
I20260924 04:11:04.370041 14221 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:11:04.372576 14236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:11:04.373903 14221 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:11:04.374140 14221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-0/wal
uuid: "a808f529633040fa88cd3449dd7017dc"
format_stamp: "Formatted at 2026-09-24 04:11:04 on dist-test-slave-00ld"
server_key: "c802f9bf344a5ec4b2682c6437d62edd"
server_key_iv: "1e3005a93af7b3af26bb524be11f2c12"
server_key_version: "encryptionkey@0"
I20260924 04:11:04.374253 14221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:11:04.384076 14221 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:11:04.386868 14221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:11:04.387068 14221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:11:04.387639 14221 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:11:04.388437 14221 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:11:04.388513 14221 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:11:04.388604 14221 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:11:04.388653 14221 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:11:04.397514 14221 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37167
I20260924 04:11:04.397533 14349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37167 every 8 connection(s)
I20260924 04:11:04.398536 14221 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:11:04.406503 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 14221
I20260924 04:11:04.406674 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:11:04.407042 392 external_mini_cluster.cc:1428] Setting key e228d3951e6074ee9842064e1dfc04f7
Sep 24 04:11:04 dist-test-slave-00ld krb5kdc[14114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223064, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:11:04.414091 14198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:04.400598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51103 (local address 127.0.98.62:36317)
0924 04:11:04.400768 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 04:11:04.400772 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:11:04.402089 (+ 1317us) server_negotiation.cc:407] Connection header received
0924 04:11:04.403162 (+ 1073us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:11:04.403166 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:11:04.403230 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:11:04.403344 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:11:04.404730 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:04.405281 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:04.406644 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:04.407063 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:04.407196 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:11:04.410361 (+ 3165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:11:04.410365 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:11:04.410367 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:11:04.410402 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:11:04.411885 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:11:04.412422 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:11:04.412427 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:11:04.412429 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:11:04.412545 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:11:04.412933 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:11:04.412938 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:11:04.412940 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:11:04.413132 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:11:04.413239 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 04:11:04.413794 (+ 555us) server_negotiation.cc:299] Negotiation successful
0924 04:11:04.413925 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 04:11:04.414970 14352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:04.400965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36317 (local address 127.0.98.1:51103)
0924 04:11:04.401915 (+ 950us) negotiation.cc:107] Waiting for socket to connect
0924 04:11:04.401961 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 04:11:04.402905 (+ 944us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:11:04.403538 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:11:04.403546 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:11:04.403992 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:11:04.404555 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:11:04.404570 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:11:04.405417 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:11:04.405420 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:11:04.406442 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:11:04.406456 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:11:04.407302 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:11:04.407307 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:11:04.407534 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:11:04.408238 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:11:04.408279 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:11:04.410183 (+ 1904us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:11:04.412017 (+ 1834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:11:04.412024 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:11:04.412043 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:11:04.412317 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:11:04.412642 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:11:04.412646 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:11:04.412648 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:11:04.412837 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:11:04.413249 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:11:04.413258 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:11:04.413529 (+ 271us) client_negotiation.cc:769] Sending connection context
0924 04:11:04.413741 (+ 212us) client_negotiation.cc:240] Negotiation successful
0924 04:11:04.414106 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":732,"thread_start_us":564,"threads_started":1}
I20260924 04:11:04.416311 14350 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36317
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 04:11:04.416616 14350 heartbeater.cc:464] Registering TS with master...
I20260924 04:11:04.417155 14350 heartbeater.cc:511] Master 127.0.98.62:36317 requested a full tablet report, sending...
I20260924 04:11:04.418694 14159 ts_manager.cc:194] Registered new tserver with Master: a808f529633040fa88cd3449dd7017dc (127.0.98.1:37167)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 04:11:04.419891 14159 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:51103
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:11:04.434975 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36317
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:11:04.544783 14355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:11:04.545073 14355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:11:04.545117 14355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:11:04.549172 14355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:11:04.549230 14355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:11:04.549288 14355 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:11:04.554934 14355 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36317
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14355
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:11:04.556092 14355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:11:04.557191 14355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:11:04.567023 14361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:11:04.567045 14360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:11:04.567056 14363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:11:04.567302 14355 server_base.cc:1034] running on GCE node
I20260924 04:11:04.567804 14355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:11:04.568490 14355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:11:04.569674 14355 hybrid_clock.cc:648] HybridClock initialized: now 1790223064569657 us; error 48 us; skew 500 ppm
Sep 24 04:11:04 dist-test-slave-00ld krb5kdc[14114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223064, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:11:04.572336 14355 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:11:04.573284 14355 webserver.cc:460] Webserver started at http://127.0.98.2:37241/ using document root <none> and password file <none>
I20260924 04:11:04.573844 14355 fs_manager.cc:359] Metadata directory not provided
I20260924 04:11:04.573935 14355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:11:04.574172 14355 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:11:04.576195 14355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-1/data/instance:
uuid: "b57c26125c704ebf960138690cb649c3"
format_stamp: "Formatted at 2026-09-24 04:11:04 on dist-test-slave-00ld"
server_key: "7e9bf5bf9dc9dd220b3f395f38ae55ae"
server_key_iv: "c7bcbd016f811948b6457e1b3de06e63"
server_key_version: "encryptionkey@0"
I20260924 04:11:04.576820 14355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b57c26125c704ebf960138690cb649c3"
format_stamp: "Formatted at 2026-09-24 04:11:04 on dist-test-slave-00ld"
server_key: "7e9bf5bf9dc9dd220b3f395f38ae55ae"
server_key_iv: "c7bcbd016f811948b6457e1b3de06e63"
server_key_version: "encryptionkey@0"
I20260924 04:11:04.580479 14355 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:11:04.582857 14370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:11:04.584075 14355 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:11:04.584210 14355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-1/wal
uuid: "b57c26125c704ebf960138690cb649c3"
format_stamp: "Formatted at 2026-09-24 04:11:04 on dist-test-slave-00ld"
server_key: "7e9bf5bf9dc9dd220b3f395f38ae55ae"
server_key_iv: "c7bcbd016f811948b6457e1b3de06e63"
server_key_version: "encryptionkey@0"
I20260924 04:11:04.584317 14355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:11:04.608844 14355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:11:04.611954 14355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:11:04.612171 14355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:11:04.612819 14355 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:11:04.613641 14355 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:11:04.613719 14355 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:11:04.613807 14355 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:11:04.613857 14355 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:11:04.622773 14355 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34039
I20260924 04:11:04.622800 14483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34039 every 8 connection(s)
I20260924 04:11:04.623639 14355 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:11:04.631641 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 14355
I20260924 04:11:04.631775 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:11:04.632054 392 external_mini_cluster.cc:1428] Setting key 54b1df95b7e3f7082115137512847f84
Sep 24 04:11:04 dist-test-slave-00ld krb5kdc[14114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223064, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:11:04.637948 14198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:04.625589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54735 (local address 127.0.98.62:36317)
0924 04:11:04.625732 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 04:11:04.625736 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:11:04.626730 (+ 994us) server_negotiation.cc:407] Connection header received
0924 04:11:04.627703 (+ 973us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:11:04.627707 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:11:04.627766 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:11:04.627870 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:11:04.629078 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:04.629619 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:04.630766 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:04.631003 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:04.631128 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:11:04.633761 (+ 2633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:11:04.633765 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:11:04.633767 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:11:04.633795 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:11:04.635526 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:11:04.636172 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:11:04.636174 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:11:04.636175 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:11:04.636249 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:11:04.636656 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:11:04.636658 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:11:04.636659 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:11:04.636951 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:11:04.637054 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 04:11:04.637536 (+ 482us) server_negotiation.cc:299] Negotiation successful
0924 04:11:04.637682 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 04:11:04.638566 14486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:04.625872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36317 (local address 127.0.98.2:54735)
0924 04:11:04.626570 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0924 04:11:04.626612 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 04:11:04.627485 (+ 873us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:11:04.628049 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:11:04.628056 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:11:04.628428 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:11:04.628927 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:11:04.628941 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:11:04.629746 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:11:04.629757 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:11:04.630638 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:11:04.630647 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:11:04.631171 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:11:04.631175 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:11:04.631345 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:11:04.631961 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:11:04.631977 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:11:04.633597 (+ 1620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:11:04.635673 (+ 2076us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:11:04.635681 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:11:04.635700 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:11:04.636067 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:11:04.636339 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:11:04.636343 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:11:04.636348 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:11:04.636563 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:11:04.637058 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:11:04.637063 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:11:04.637292 (+ 229us) client_negotiation.cc:769] Sending connection context
0924 04:11:04.637451 (+ 159us) client_negotiation.cc:240] Negotiation successful
0924 04:11:04.637707 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":127,"threads_started":1}
I20260924 04:11:04.639780 14484 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36317
I20260924 04:11:04.640074 14484 heartbeater.cc:464] Registering TS with master...
I20260924 04:11:04.640720 14484 heartbeater.cc:511] Master 127.0.98.62:36317 requested a full tablet report, sending...
I20260924 04:11:04.641861 14159 ts_manager.cc:194] Registered new tserver with Master: b57c26125c704ebf960138690cb649c3 (127.0.98.2:34039)
I20260924 04:11:04.642446 14159 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:54735
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:11:04.660259 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36317
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:11:04.769830 14489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:11:04.770073 14489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:11:04.770115 14489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:11:04.774175 14489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:11:04.774232 14489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:11:04.774315 14489 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:11:04.779745 14489 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36317
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14489
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:11:04.780853 14489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:11:04.781860 14489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:11:04.791383 14495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:11:04.791394 14494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:11:04.791394 14497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:11:04.791483 14489 server_base.cc:1034] running on GCE node
I20260924 04:11:04.791988 14489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:11:04.792623 14489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:11:04.793828 14489 hybrid_clock.cc:648] HybridClock initialized: now 1790223064793804 us; error 35 us; skew 500 ppm
Sep 24 04:11:04 dist-test-slave-00ld krb5kdc[14114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223064, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:11:04.796569 14489 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:11:04.797453 14489 webserver.cc:460] Webserver started at http://127.0.98.3:41653/ using document root <none> and password file <none>
I20260924 04:11:04.798005 14489 fs_manager.cc:359] Metadata directory not provided
I20260924 04:11:04.798094 14489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:11:04.798323 14489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:11:04.800318 14489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-2/data/instance:
uuid: "c68b0618084a48a6aa96560ea5cf20f7"
format_stamp: "Formatted at 2026-09-24 04:11:04 on dist-test-slave-00ld"
server_key: "d12377f81f2dd5b855781d00330550c9"
server_key_iv: "3575e29b7a2156dad63ff1b610294213"
server_key_version: "encryptionkey@0"
I20260924 04:11:04.800918 14489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c68b0618084a48a6aa96560ea5cf20f7"
format_stamp: "Formatted at 2026-09-24 04:11:04 on dist-test-slave-00ld"
server_key: "d12377f81f2dd5b855781d00330550c9"
server_key_iv: "3575e29b7a2156dad63ff1b610294213"
server_key_version: "encryptionkey@0"
I20260924 04:11:04.804818 14489 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 04:11:04.807147 14504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:11:04.808441 14489 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:11:04.808574 14489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-2/wal
uuid: "c68b0618084a48a6aa96560ea5cf20f7"
format_stamp: "Formatted at 2026-09-24 04:11:04 on dist-test-slave-00ld"
server_key: "d12377f81f2dd5b855781d00330550c9"
server_key_iv: "3575e29b7a2156dad63ff1b610294213"
server_key_version: "encryptionkey@0"
I20260924 04:11:04.808686 14489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:11:04.828245 14489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:11:04.831219 14489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:11:04.831488 14489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:11:04.832091 14489 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:11:04.832975 14489 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:11:04.833042 14489 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:11:04.833128 14489 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:11:04.833179 14489 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:11:04.842113 14489 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45681
I20260924 04:11:04.842136 14617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45681 every 8 connection(s)
I20260924 04:11:04.843047 14489 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:11:04.846246 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 14489
I20260924 04:11:04.846377 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:11:04.846689 392 external_mini_cluster.cc:1428] Setting key fb095dd23507ff927f52372a192f7ae3
Sep 24 04:11:04 dist-test-slave-00ld krb5kdc[14114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223064, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:11:04.857318 14198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:04.845089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41593 (local address 127.0.98.62:36317)
0924 04:11:04.845250 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 04:11:04.845258 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 04:11:04.846547 (+ 1289us) server_negotiation.cc:407] Connection header received
0924 04:11:04.847695 (+ 1148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:11:04.847698 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:11:04.847792 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:11:04.847897 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:11:04.849398 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:04.849916 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:04.851002 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:04.851213 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:04.851304 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:11:04.853759 (+ 2455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:11:04.853765 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:11:04.853773 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:11:04.853796 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:11:04.855120 (+ 1324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:11:04.855720 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:11:04.855725 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:11:04.855727 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:11:04.855812 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:11:04.856152 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:11:04.856157 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:11:04.856159 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:11:04.856341 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:11:04.856449 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 04:11:04.857018 (+ 569us) server_negotiation.cc:299] Negotiation successful
0924 04:11:04.857147 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 04:11:04.858098 14620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:04.845565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36317 (local address 127.0.98.3:41593)
0924 04:11:04.846364 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0924 04:11:04.846425 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0924 04:11:04.847439 (+ 1014us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:11:04.848125 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:11:04.848135 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:11:04.848546 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:11:04.849258 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:11:04.849274 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:11:04.850027 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:11:04.850029 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:11:04.850892 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:11:04.850900 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:11:04.851347 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:11:04.851350 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:11:04.851486 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:11:04.852059 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:11:04.852075 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:11:04.853530 (+ 1455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:11:04.855237 (+ 1707us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:11:04.855243 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:11:04.855258 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:11:04.855617 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:11:04.855905 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:11:04.855909 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:11:04.855911 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:11:04.856055 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:11:04.856446 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:11:04.856453 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:11:04.856767 (+ 314us) client_negotiation.cc:769] Sending connection context
0924 04:11:04.856966 (+ 199us) client_negotiation.cc:240] Negotiation successful
0924 04:11:04.857282 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":127,"threads_started":1}
I20260924 04:11:04.859189 14618 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36317
I20260924 04:11:04.859453 14618 heartbeater.cc:464] Registering TS with master...
I20260924 04:11:04.860077 14618 heartbeater.cc:511] Master 127.0.98.62:36317 requested a full tablet report, sending...
I20260924 04:11:04.861196 14159 ts_manager.cc:194] Registered new tserver with Master: c68b0618084a48a6aa96560ea5cf20f7 (127.0.98.3:45681)
I20260924 04:11:04.861728 14159 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41593
I20260924 04:11:04.872907 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:11:04.882193 14198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:04.874878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56332 (local address 127.0.98.62:36317)
0924 04:11:04.874982 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0924 04:11:04.874986 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:11:04.875242 (+ 256us) server_negotiation.cc:407] Connection header received
0924 04:11:04.875409 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:11:04.875412 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:11:04.875466 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:11:04.875554 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:11:04.876280 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:04.876887 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:04.877740 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:04.877978 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:04.878094 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:11:04.879478 (+ 1384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:11:04.879481 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:11:04.879483 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:11:04.879505 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:11:04.880792 (+ 1287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:11:04.881168 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:11:04.881171 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:11:04.881172 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:11:04.881244 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:11:04.881496 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:11:04.881498 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:11:04.881500 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:11:04.881706 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:11:04.881805 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 04:11:04.881979 (+ 174us) server_negotiation.cc:299] Negotiation successful
0924 04:11:04.882075 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 04:11:04.884899 14159 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56332:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:11:04.887924 14159 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:11:04.902547 14630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:04.896407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58532 (local address 127.0.98.1:37167)
0924 04:11:04.896837 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0924 04:11:04.896852 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 04:11:04.896875 (+ 23us) server_negotiation.cc:407] Connection header received
0924 04:11:04.897088 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:11:04.897097 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:11:04.897324 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:11:04.897481 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:11:04.897934 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:04.899043 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:04.900586 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:04.901306 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:04.901406 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:11:04.901556 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 04:11:04.902284 (+ 728us) server_negotiation.cc:299] Negotiation successful
0924 04:11:04.902390 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":63,"threads_started":1}
I20260924 04:11:04.902567 14633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:04.897147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46162 (local address 127.0.98.2:34039)
0924 04:11:04.897892 (+ 745us) server_negotiation.cc:206] Beginning negotiation
0924 04:11:04.897905 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 04:11:04.897930 (+ 25us) server_negotiation.cc:407] Connection header received
0924 04:11:04.898007 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:11:04.898014 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:11:04.898170 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:11:04.898311 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:11:04.898541 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:04.899539 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:04.901033 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:04.901790 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:04.901904 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:11:04.902028 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0924 04:11:04.902380 (+ 352us) server_negotiation.cc:299] Negotiation successful
0924 04:11:04.902456 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":637,"thread_start_us":81,"threads_started":1}
I20260924 04:11:04.902820 14634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:04.897491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58884 (local address 127.0.98.3:45681)
0924 04:11:04.898140 (+ 649us) server_negotiation.cc:206] Beginning negotiation
0924 04:11:04.898152 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 04:11:04.898174 (+ 22us) server_negotiation.cc:407] Connection header received
0924 04:11:04.898245 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:11:04.898250 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:11:04.898422 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:11:04.898554 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:11:04.898930 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:04.900074 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:04.901362 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:04.901932 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:04.902123 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:11:04.902263 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 04:11:04.902489 (+ 226us) server_negotiation.cc:299] Negotiation successful
0924 04:11:04.902597 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":133,"threads_started":1}
I20260924 04:11:04.902912 14631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:04.896641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34039 (local address 127.0.0.1:46162)
0924 04:11:04.897538 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0924 04:11:04.897553 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:11:04.897647 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:11:04.898292 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:11:04.898294 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:11:04.898310 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:11:04.898418 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:11:04.898427 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:11:04.899694 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:11:04.899697 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:11:04.900911 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:11:04.900921 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:11:04.901920 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:11:04.901923 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:11:04.902032 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:11:04.902067 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 04:11:04.902135 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 04:11:04.902233 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":839,"mutex_wait_us":1,"thread_start_us":125,"threads_started":1}
I20260924 04:11:04.903113 14632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:04.897065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45681 (local address 127.0.0.1:58884)
0924 04:11:04.897877 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0924 04:11:04.897891 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:11:04.897974 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:11:04.898705 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:11:04.898707 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:11:04.898724 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:11:04.898805 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:11:04.898810 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:11:04.900218 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:11:04.900221 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:11:04.901242 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:11:04.901249 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:11:04.902132 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:11:04.902136 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:11:04.902264 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:11:04.902286 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 04:11:04.902358 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 04:11:04.902420 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":744,"mutex_wait_us":22,"thread_start_us":83,"threads_started":1}
I20260924 04:11:04.902177 14629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:04.896289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37167 (local address 127.0.0.1:58532)
0924 04:11:04.896671 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0924 04:11:04.896744 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0924 04:11:04.896881 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:11:04.897697 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:11:04.897702 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:11:04.897720 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:11:04.897817 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:11:04.897822 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:11:04.899162 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:11:04.899166 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:11:04.900463 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:11:04.900472 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:11:04.901624 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:11:04.901628 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:11:04.901735 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:11:04.901781 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 04:11:04.901884 (+ 103us) client_negotiation.cc:240] Negotiation successful
0924 04:11:04.902005 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"mutex_wait_us":1,"thread_start_us":96,"threads_started":1}
I20260924 04:11:04.905946 14284 tablet_service.cc:1467] Processing CreateTablet for tablet 7a824eb115b64f23a7afae2d6a3a1d3f (DEFAULT_TABLE table=test-table [id=e6bd0a45223b4ebc8ce2adf8c797018d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:11:04.906162 14418 tablet_service.cc:1467] Processing CreateTablet for tablet 7a824eb115b64f23a7afae2d6a3a1d3f (DEFAULT_TABLE table=test-table [id=e6bd0a45223b4ebc8ce2adf8c797018d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:11:04.906977 14284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a824eb115b64f23a7afae2d6a3a1d3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:11:04.907130 14418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a824eb115b64f23a7afae2d6a3a1d3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:11:04.908260 14552 tablet_service.cc:1467] Processing CreateTablet for tablet 7a824eb115b64f23a7afae2d6a3a1d3f (DEFAULT_TABLE table=test-table [id=e6bd0a45223b4ebc8ce2adf8c797018d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:11:04.909230 14552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a824eb115b64f23a7afae2d6a3a1d3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:11:04.913506 14635 tablet_bootstrap.cc:492] T 7a824eb115b64f23a7afae2d6a3a1d3f P a808f529633040fa88cd3449dd7017dc: Bootstrap starting.
I20260924 04:11:04.915813 14635 tablet_bootstrap.cc:654] T 7a824eb115b64f23a7afae2d6a3a1d3f P a808f529633040fa88cd3449dd7017dc: Neither blocks nor log segments found. Creating new log.
I20260924 04:11:04.915838 14637 tablet_bootstrap.cc:492] T 7a824eb115b64f23a7afae2d6a3a1d3f P c68b0618084a48a6aa96560ea5cf20f7: Bootstrap starting.
I20260924 04:11:04.916388 14636 tablet_bootstrap.cc:492] T 7a824eb115b64f23a7afae2d6a3a1d3f P b57c26125c704ebf960138690cb649c3: Bootstrap starting.
I20260924 04:11:04.917315 14635 log.cc:824] T 7a824eb115b64f23a7afae2d6a3a1d3f P a808f529633040fa88cd3449dd7017dc: Log is configured to *not* fsync() on all Append() calls
I20260924 04:11:04.917997 14637 tablet_bootstrap.cc:654] T 7a824eb115b64f23a7afae2d6a3a1d3f P c68b0618084a48a6aa96560ea5cf20f7: Neither blocks nor log segments found. Creating new log.
I20260924 04:11:04.918898 14636 tablet_bootstrap.cc:654] T 7a824eb115b64f23a7afae2d6a3a1d3f P b57c26125c704ebf960138690cb649c3: Neither blocks nor log segments found. Creating new log.
I20260924 04:11:04.919070 14635 tablet_bootstrap.cc:492] T 7a824eb115b64f23a7afae2d6a3a1d3f P a808f529633040fa88cd3449dd7017dc: No bootstrap required, opened a new log
I20260924 04:11:04.919281 14635 ts_tablet_manager.cc:1397] T 7a824eb115b64f23a7afae2d6a3a1d3f P a808f529633040fa88cd3449dd7017dc: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 04:11:04.919324 14637 log.cc:824] T 7a824eb115b64f23a7afae2d6a3a1d3f P c68b0618084a48a6aa96560ea5cf20f7: Log is configured to *not* fsync() on all Append() calls
I20260924 04:11:04.919662 14636 log.cc:824] T 7a824eb115b64f23a7afae2d6a3a1d3f P b57c26125c704ebf960138690cb649c3: Log is configured to *not* fsync() on all Append() calls
I20260924 04:11:04.921084 14637 tablet_bootstrap.cc:492] T 7a824eb115b64f23a7afae2d6a3a1d3f P c68b0618084a48a6aa96560ea5cf20f7: No bootstrap required, opened a new log
I20260924 04:11:04.921291 14637 ts_tablet_manager.cc:1397] T 7a824eb115b64f23a7afae2d6a3a1d3f P c68b0618084a48a6aa96560ea5cf20f7: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 04:11:04.922335 14636 tablet_bootstrap.cc:492] T 7a824eb115b64f23a7afae2d6a3a1d3f P b57c26125c704ebf960138690cb649c3: No bootstrap required, opened a new log
I20260924 04:11:04.922495 14636 ts_tablet_manager.cc:1397] T 7a824eb115b64f23a7afae2d6a3a1d3f P b57c26125c704ebf960138690cb649c3: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 04:11:04.923434 14635 raft_consensus.cc:348] T 7a824eb115b64f23a7afae2d6a3a1d3f P a808f529633040fa88cd3449dd7017dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a808f529633040fa88cd3449dd7017dc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37167 } } peers { permanent_uuid: "c68b0618084a48a6aa96560ea5cf20f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45681 } } peers { permanent_uuid: "b57c26125c704ebf960138690cb649c3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34039 } }
I20260924 04:11:04.923635 14635 raft_consensus.cc:374] T 7a824eb115b64f23a7afae2d6a3a1d3f P a808f529633040fa88cd3449dd7017dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:11:04.923666 14635 raft_consensus.cc:729] T 7a824eb115b64f23a7afae2d6a3a1d3f P a808f529633040fa88cd3449dd7017dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a808f529633040fa88cd3449dd7017dc, State: Initialized, Role: FOLLOWER
I20260924 04:11:04.924070 14635 consensus_queue.cc:260] T 7a824eb115b64f23a7afae2d6a3a1d3f P a808f529633040fa88cd3449dd7017dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a808f529633040fa88cd3449dd7017dc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37167 } } peers { permanent_uuid: "c68b0618084a48a6aa96560ea5cf20f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45681 } } peers { permanent_uuid: "b57c26125c704ebf960138690cb649c3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34039 } }
I20260924 04:11:04.924665 14350 heartbeater.cc:503] Master 127.0.98.62:36317 was elected leader, sending a full tablet report...
I20260924 04:11:04.925071 14635 ts_tablet_manager.cc:1428] T 7a824eb115b64f23a7afae2d6a3a1d3f P a808f529633040fa88cd3449dd7017dc: Time spent starting tablet: real 0.006s user 0.002s sys 0.005s
I20260924 04:11:04.925872 14637 raft_consensus.cc:348] T 7a824eb115b64f23a7afae2d6a3a1d3f P c68b0618084a48a6aa96560ea5cf20f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a808f529633040fa88cd3449dd7017dc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37167 } } peers { permanent_uuid: "c68b0618084a48a6aa96560ea5cf20f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45681 } } peers { permanent_uuid: "b57c26125c704ebf960138690cb649c3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34039 } }
I20260924 04:11:04.926118 14637 raft_consensus.cc:374] T 7a824eb115b64f23a7afae2d6a3a1d3f P c68b0618084a48a6aa96560ea5cf20f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:11:04.926172 14637 raft_consensus.cc:729] T 7a824eb115b64f23a7afae2d6a3a1d3f P c68b0618084a48a6aa96560ea5cf20f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c68b0618084a48a6aa96560ea5cf20f7, State: Initialized, Role: FOLLOWER
I20260924 04:11:04.926187 14636 raft_consensus.cc:348] T 7a824eb115b64f23a7afae2d6a3a1d3f P b57c26125c704ebf960138690cb649c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a808f529633040fa88cd3449dd7017dc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37167 } } peers { permanent_uuid: "c68b0618084a48a6aa96560ea5cf20f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45681 } } peers { permanent_uuid: "b57c26125c704ebf960138690cb649c3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34039 } }
I20260924 04:11:04.926431 14636 raft_consensus.cc:374] T 7a824eb115b64f23a7afae2d6a3a1d3f P b57c26125c704ebf960138690cb649c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:11:04.926491 14636 raft_consensus.cc:729] T 7a824eb115b64f23a7afae2d6a3a1d3f P b57c26125c704ebf960138690cb649c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b57c26125c704ebf960138690cb649c3, State: Initialized, Role: FOLLOWER
I20260924 04:11:04.926620 14637 consensus_queue.cc:260] T 7a824eb115b64f23a7afae2d6a3a1d3f P c68b0618084a48a6aa96560ea5cf20f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a808f529633040fa88cd3449dd7017dc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37167 } } peers { permanent_uuid: "c68b0618084a48a6aa96560ea5cf20f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45681 } } peers { permanent_uuid: "b57c26125c704ebf960138690cb649c3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34039 } }
I20260924 04:11:04.926870 14636 consensus_queue.cc:260] T 7a824eb115b64f23a7afae2d6a3a1d3f P b57c26125c704ebf960138690cb649c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a808f529633040fa88cd3449dd7017dc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37167 } } peers { permanent_uuid: "c68b0618084a48a6aa96560ea5cf20f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45681 } } peers { permanent_uuid: "b57c26125c704ebf960138690cb649c3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34039 } }
I20260924 04:11:04.927300 14618 heartbeater.cc:503] Master 127.0.98.62:36317 was elected leader, sending a full tablet report...
I20260924 04:11:04.927811 14636 ts_tablet_manager.cc:1428] T 7a824eb115b64f23a7afae2d6a3a1d3f P b57c26125c704ebf960138690cb649c3: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260924 04:11:04.928166 14484 heartbeater.cc:503] Master 127.0.98.62:36317 was elected leader, sending a full tablet report...
I20260924 04:11:04.930027 14637 ts_tablet_manager.cc:1428] T 7a824eb115b64f23a7afae2d6a3a1d3f P c68b0618084a48a6aa96560ea5cf20f7: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260924 04:11:04.965754 14642 raft_consensus.cc:482] T 7a824eb115b64f23a7afae2d6a3a1d3f P c68b0618084a48a6aa96560ea5cf20f7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:11:04.965950 14642 raft_consensus.cc:504] T 7a824eb115b64f23a7afae2d6a3a1d3f P c68b0618084a48a6aa96560ea5cf20f7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a808f529633040fa88cd3449dd7017dc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37167 } } peers { permanent_uuid: "c68b0618084a48a6aa96560ea5cf20f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45681 } } peers { permanent_uuid: "b57c26125c704ebf960138690cb649c3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34039 } }
I20260924 04:11:04.967038 14642 leader_election.cc:288] T 7a824eb115b64f23a7afae2d6a3a1d3f P c68b0618084a48a6aa96560ea5cf20f7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a808f529633040fa88cd3449dd7017dc (127.0.98.1:37167), b57c26125c704ebf960138690cb649c3 (127.0.98.2:34039)
I20260924 04:11:04.972460 14630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:04.967455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39149 (local address 127.0.98.1:37167)
0924 04:11:04.967582 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 04:11:04.967586 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:11:04.967970 (+ 384us) server_negotiation.cc:407] Connection header received
0924 04:11:04.968058 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:11:04.968060 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:11:04.968103 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:11:04.968165 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:11:04.968418 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:04.969230 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:04.971018 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:04.971724 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:04.971832 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:11:04.971909 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 04:11:04.972225 (+ 316us) server_negotiation.cc:299] Negotiation successful
0924 04:11:04.972309 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 04:11:04.972451 14633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:04.967500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54415 (local address 127.0.98.2:34039)
0924 04:11:04.967653 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0924 04:11:04.967661 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 04:11:04.967680 (+ 19us) server_negotiation.cc:407] Connection header received
0924 04:11:04.967740 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:11:04.967744 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:11:04.967805 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:11:04.967892 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:11:04.968254 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:04.969180 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:04.971018 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:04.971698 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:04.971845 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:11:04.971908 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 04:11:04.972225 (+ 317us) server_negotiation.cc:299] Negotiation successful
0924 04:11:04.972308 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 04:11:04.972494 14620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:04.967341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34039 (local address 127.0.98.3:54415)
0924 04:11:04.967486 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 04:11:04.967503 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 04:11:04.967607 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:11:04.967925 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:11:04.967931 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:11:04.967956 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:11:04.968111 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:11:04.968117 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:11:04.969305 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:11:04.969309 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:11:04.970875 (+ 1566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:11:04.970886 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:11:04.971846 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:11:04.971849 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:11:04.972062 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:11:04.972112 (+ 50us) client_negotiation.cc:769] Sending connection context
0924 04:11:04.972191 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 04:11:04.972329 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":252}
I20260924 04:11:04.972509 14644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:04.967341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37167 (local address 127.0.98.3:39149)
0924 04:11:04.967845 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0924 04:11:04.967855 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 04:11:04.967970 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:11:04.968195 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:11:04.968198 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:11:04.968218 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:11:04.968312 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:11:04.968316 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:11:04.969322 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:11:04.969324 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:11:04.970875 (+ 1551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:11:04.970886 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:11:04.971846 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:11:04.971849 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:11:04.972042 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:11:04.972106 (+ 64us) client_negotiation.cc:769] Sending connection context
0924 04:11:04.972187 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 04:11:04.972329 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"mutex_wait_us":391,"spinlock_wait_cycles":9600,"thread_start_us":147,"threads_started":1}
I20260924 04:11:04.973264 14304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a824eb115b64f23a7afae2d6a3a1d3f" candidate_uuid: "c68b0618084a48a6aa96560ea5cf20f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a808f529633040fa88cd3449dd7017dc" is_pre_election: true
I20260924 04:11:04.973582 14438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a824eb115b64f23a7afae2d6a3a1d3f" candidate_uuid: "c68b0618084a48a6aa96560ea5cf20f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b57c26125c704ebf960138690cb649c3" is_pre_election: true
I20260924 04:11:04.973624 14304 raft_consensus.cc:2454] T 7a824eb115b64f23a7afae2d6a3a1d3f P a808f529633040fa88cd3449dd7017dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c68b0618084a48a6aa96560ea5cf20f7 in term 0.
I20260924 04:11:04.973879 14438 raft_consensus.cc:2454] T 7a824eb115b64f23a7afae2d6a3a1d3f P b57c26125c704ebf960138690cb649c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c68b0618084a48a6aa96560ea5cf20f7 in term 0.
I20260924 04:11:04.974139 14508 leader_election.cc:302] T 7a824eb115b64f23a7afae2d6a3a1d3f P c68b0618084a48a6aa96560ea5cf20f7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a808f529633040fa88cd3449dd7017dc, c68b0618084a48a6aa96560ea5cf20f7; no voters:
I20260924 04:11:04.974620 14642 raft_consensus.cc:2781] T 7a824eb115b64f23a7afae2d6a3a1d3f P c68b0618084a48a6aa96560ea5cf20f7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:11:04.974713 14642 raft_consensus.cc:482] T 7a824eb115b64f23a7afae2d6a3a1d3f P c68b0618084a48a6aa96560ea5cf20f7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:11:04.974774 14642 raft_consensus.cc:3037] T 7a824eb115b64f23a7afae2d6a3a1d3f P c68b0618084a48a6aa96560ea5cf20f7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:11:04.975814 14642 raft_consensus.cc:504] T 7a824eb115b64f23a7afae2d6a3a1d3f P c68b0618084a48a6aa96560ea5cf20f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a808f529633040fa88cd3449dd7017dc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37167 } } peers { permanent_uuid: "c68b0618084a48a6aa96560ea5cf20f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45681 } } peers { permanent_uuid: "b57c26125c704ebf960138690cb649c3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34039 } }
I20260924 04:11:04.976187 14642 leader_election.cc:288] T 7a824eb115b64f23a7afae2d6a3a1d3f P c68b0618084a48a6aa96560ea5cf20f7 [CANDIDATE]: Term 1 election: Requested vote from peers a808f529633040fa88cd3449dd7017dc (127.0.98.1:37167), b57c26125c704ebf960138690cb649c3 (127.0.98.2:34039)
I20260924 04:11:04.976564 14304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a824eb115b64f23a7afae2d6a3a1d3f" candidate_uuid: "c68b0618084a48a6aa96560ea5cf20f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a808f529633040fa88cd3449dd7017dc"
I20260924 04:11:04.976693 14438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a824eb115b64f23a7afae2d6a3a1d3f" candidate_uuid: "c68b0618084a48a6aa96560ea5cf20f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b57c26125c704ebf960138690cb649c3"
I20260924 04:11:04.976732 14304 raft_consensus.cc:3037] T 7a824eb115b64f23a7afae2d6a3a1d3f P a808f529633040fa88cd3449dd7017dc [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:11:04.976821 14438 raft_consensus.cc:3037] T 7a824eb115b64f23a7afae2d6a3a1d3f P b57c26125c704ebf960138690cb649c3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:11:04.977846 14304 raft_consensus.cc:2454] T 7a824eb115b64f23a7afae2d6a3a1d3f P a808f529633040fa88cd3449dd7017dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c68b0618084a48a6aa96560ea5cf20f7 in term 1.
I20260924 04:11:04.978024 14438 raft_consensus.cc:2454] T 7a824eb115b64f23a7afae2d6a3a1d3f P b57c26125c704ebf960138690cb649c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c68b0618084a48a6aa96560ea5cf20f7 in term 1.
I20260924 04:11:04.978137 14508 leader_election.cc:302] T 7a824eb115b64f23a7afae2d6a3a1d3f P c68b0618084a48a6aa96560ea5cf20f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a808f529633040fa88cd3449dd7017dc, c68b0618084a48a6aa96560ea5cf20f7; no voters:
I20260924 04:11:04.978312 14642 raft_consensus.cc:2781] T 7a824eb115b64f23a7afae2d6a3a1d3f P c68b0618084a48a6aa96560ea5cf20f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:11:04.978528 14642 raft_consensus.cc:686] T 7a824eb115b64f23a7afae2d6a3a1d3f P c68b0618084a48a6aa96560ea5cf20f7 [term 1 LEADER]: Becoming Leader. State: Replica: c68b0618084a48a6aa96560ea5cf20f7, State: Running, Role: LEADER
I20260924 04:11:04.978878 14642 consensus_queue.cc:237] T 7a824eb115b64f23a7afae2d6a3a1d3f P c68b0618084a48a6aa96560ea5cf20f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a808f529633040fa88cd3449dd7017dc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37167 } } peers { permanent_uuid: "c68b0618084a48a6aa96560ea5cf20f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45681 } } peers { permanent_uuid: "b57c26125c704ebf960138690cb649c3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34039 } }
I20260924 04:11:04.981961 14158 catalog_manager.cc:5495] T 7a824eb115b64f23a7afae2d6a3a1d3f P c68b0618084a48a6aa96560ea5cf20f7 reported cstate change: term changed from 0 to 1, leader changed from <none> to c68b0618084a48a6aa96560ea5cf20f7 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "c68b0618084a48a6aa96560ea5cf20f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a808f529633040fa88cd3449dd7017dc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c68b0618084a48a6aa96560ea5cf20f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b57c26125c704ebf960138690cb649c3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34039 } health_report { overall_health: UNKNOWN } } }
I20260924 04:11:05.005802 14634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:05.001897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58898 (local address 127.0.98.3:45681)
0924 04:11:05.002056 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 04:11:05.002061 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:11:05.002076 (+ 15us) server_negotiation.cc:407] Connection header received
0924 04:11:05.002244 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:11:05.002246 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:11:05.002304 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:11:05.002416 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:11:05.002674 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:05.003172 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:05.004112 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:05.004476 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:05.004621 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:11:05.004979 (+ 358us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:11:05.005403 (+ 424us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:11:05.005543 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 04:11:05.005583 (+ 40us) server_negotiation.cc:299] Negotiation successful
0924 04:11:05.005637 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 04:11:05.011605 14304 raft_consensus.cc:1261] T 7a824eb115b64f23a7afae2d6a3a1d3f P a808f529633040fa88cd3449dd7017dc [term 1 FOLLOWER]: Refusing update from remote peer c68b0618084a48a6aa96560ea5cf20f7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:11:05.011613 14438 raft_consensus.cc:1261] T 7a824eb115b64f23a7afae2d6a3a1d3f P b57c26125c704ebf960138690cb649c3 [term 1 FOLLOWER]: Refusing update from remote peer c68b0618084a48a6aa96560ea5cf20f7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:11:05.012379 14642 consensus_queue.cc:1035] T 7a824eb115b64f23a7afae2d6a3a1d3f P c68b0618084a48a6aa96560ea5cf20f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b57c26125c704ebf960138690cb649c3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:11:05.012633 14645 consensus_queue.cc:1035] T 7a824eb115b64f23a7afae2d6a3a1d3f P c68b0618084a48a6aa96560ea5cf20f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a808f529633040fa88cd3449dd7017dc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:11:05.021246 14650 mvcc.cc:205] Tried to move back new op lower bound from 7332753674280185856 to 7332753674156322816. Current Snapshot: MvccSnapshot[applied={T|T < 7332753674280185856}]
I20260924 04:11:05.029174 14158 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56332:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:11:05.029347 14158 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56332:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:11:05.031570 14158 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 001f36501e0c44f090cf0e3ba3a05727: Sending DeleteTablet for 3 replicas of tablet 7a824eb115b64f23a7afae2d6a3a1d3f
I20260924 04:11:05.032370 14552 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a824eb115b64f23a7afae2d6a3a1d3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:11:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58884
I20260924 04:11:05.032430 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 14221
I20260924 04:11:05.032487 14418 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a824eb115b64f23a7afae2d6a3a1d3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:11:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46162
I20260924 04:11:05.033151 14658 tablet_replica.cc:330] T 7a824eb115b64f23a7afae2d6a3a1d3f P b57c26125c704ebf960138690cb649c3: stopping tablet replica
I20260924 04:11:05.033471 14658 raft_consensus.cc:2229] T 7a824eb115b64f23a7afae2d6a3a1d3f P b57c26125c704ebf960138690cb649c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:11:05.033550 14657 tablet_replica.cc:330] T 7a824eb115b64f23a7afae2d6a3a1d3f P c68b0618084a48a6aa96560ea5cf20f7: stopping tablet replica
I20260924 04:11:05.033764 14658 raft_consensus.cc:2258] T 7a824eb115b64f23a7afae2d6a3a1d3f P b57c26125c704ebf960138690cb649c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:11:05.033833 14657 raft_consensus.cc:2229] T 7a824eb115b64f23a7afae2d6a3a1d3f P c68b0618084a48a6aa96560ea5cf20f7 [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:11:05.034662 14657 raft_consensus.cc:2258] T 7a824eb115b64f23a7afae2d6a3a1d3f P c68b0618084a48a6aa96560ea5cf20f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:11:05.035758 14657 ts_tablet_manager.cc:1905] T 7a824eb115b64f23a7afae2d6a3a1d3f P c68b0618084a48a6aa96560ea5cf20f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:11:05.035800 14658 ts_tablet_manager.cc:1905] T 7a824eb115b64f23a7afae2d6a3a1d3f P b57c26125c704ebf960138690cb649c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:11:05.038054 14657 ts_tablet_manager.cc:1918] T 7a824eb115b64f23a7afae2d6a3a1d3f P c68b0618084a48a6aa96560ea5cf20f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:11:05.038121 14657 log.cc:1196] T 7a824eb115b64f23a7afae2d6a3a1d3f P c68b0618084a48a6aa96560ea5cf20f7: Deleting WAL directory at /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-2/wal/wals/7a824eb115b64f23a7afae2d6a3a1d3f
I20260924 04:11:05.038357 14657 ts_tablet_manager.cc:1939] T 7a824eb115b64f23a7afae2d6a3a1d3f P c68b0618084a48a6aa96560ea5cf20f7: Deleting consensus metadata
I20260924 04:11:05.039117 14658 ts_tablet_manager.cc:1918] T 7a824eb115b64f23a7afae2d6a3a1d3f P b57c26125c704ebf960138690cb649c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:11:05.039201 14658 log.cc:1196] T 7a824eb115b64f23a7afae2d6a3a1d3f P b57c26125c704ebf960138690cb649c3: Deleting WAL directory at /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-1/wal/wals/7a824eb115b64f23a7afae2d6a3a1d3f
I20260924 04:11:05.039283 14146 catalog_manager.cc:4841] TS c68b0618084a48a6aa96560ea5cf20f7 (127.0.98.3:45681): tablet 7a824eb115b64f23a7afae2d6a3a1d3f (table test-table [id=e6bd0a45223b4ebc8ce2adf8c797018d]) successfully deleted
I20260924 04:11:05.039526 14658 ts_tablet_manager.cc:1939] T 7a824eb115b64f23a7afae2d6a3a1d3f P b57c26125c704ebf960138690cb649c3: Deleting consensus metadata
W20260924 04:11:05.040088 14146 connection.cc:535] client connection to 127.0.98.1:37167 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260924 04:11:05.040226 14146 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 04:11:05.041009 14143 catalog_manager.cc:4841] TS b57c26125c704ebf960138690cb649c3 (127.0.98.2:34039): tablet 7a824eb115b64f23a7afae2d6a3a1d3f (table test-table [id=e6bd0a45223b4ebc8ce2adf8c797018d]) successfully deleted
I20260924 04:11:05.041232 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 14355
I20260924 04:11:05.041427 14629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:05.040915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37167 (local address 127.0.0.1:58538)
0924 04:11:05.041236 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0924 04:11:05.041309 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231}
W20260924 04:11:05.043190 14146 catalog_manager.cc:4584] TS a808f529633040fa88cd3449dd7017dc (127.0.98.1:37167): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7a824eb115b64f23a7afae2d6a3a1d3f: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37167: connect: Connection refused (error 111)
I20260924 04:11:05.048939 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 14489
I20260924 04:11:05.054831 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 14127
I20260924 04:11:05.061331 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36317
--webserver_interface=127.0.98.62
--webserver_port=35995
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36317
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:11:05.170506 14660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:11:05.170754 14660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:11:05.170794 14660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:11:05.174887 14660 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:11:05.174949 14660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:11:05.174968 14660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:11:05.175016 14660 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:11:05.175048 14660 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:11:05.180410 14660 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36317
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=35995
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14660
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:11:05.181520 14660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:11:05.182565 14660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:11:05.195562 14667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:11:05.195570 14669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:11:05.195639 14660 server_base.cc:1034] running on GCE node
W20260924 04:11:05.195559 14666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:11:05.196493 14660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:11:05.197737 14660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:11:05.198935 14660 hybrid_clock.cc:648] HybridClock initialized: now 1790223065198917 us; error 32 us; skew 500 ppm
Sep 24 04:11:05 dist-test-slave-00ld krb5kdc[14114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223065, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:11:05.201694 14660 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:11:05.202562 14660 webserver.cc:460] Webserver started at http://127.0.98.62:35995/ using document root <none> and password file <none>
I20260924 04:11:05.203090 14660 fs_manager.cc:359] Metadata directory not provided
I20260924 04:11:05.203161 14660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:11:05.206931 14660 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.002s
I20260924 04:11:05.209110 14676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:11:05.210366 14660 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:11:05.210494 14660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/master-0/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/master-0/wal
uuid: "001f36501e0c44f090cf0e3ba3a05727"
format_stamp: "Formatted at 2026-09-24 04:11:04 on dist-test-slave-00ld"
server_key: "ad4519d5f762093b834ef33571f1a933"
server_key_iv: "7c2fc7657d5745075655eafc99715370"
server_key_version: "encryptionkey@0"
I20260924 04:11:05.211163 14660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:11:05.225235 14660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:11:05.228256 14660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:11:05.228457 14660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:11:05.235507 14660 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36317
I20260924 04:11:05.235807 14728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36317 every 8 connection(s)
I20260924 04:11:05.236734 14660 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:11:05.237490 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 14660
I20260924 04:11:05.237986 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:37167
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:36317
--webserver_port=34783
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:11:05.242010 14729 sys_catalog.cc:263] Verifying existing consensus state
I20260924 04:11:05.243650 14729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 001f36501e0c44f090cf0e3ba3a05727: Bootstrap starting.
I20260924 04:11:05.260396 14729 log.cc:824] T 00000000000000000000000000000000 P 001f36501e0c44f090cf0e3ba3a05727: Log is configured to *not* fsync() on all Append() calls
I20260924 04:11:05.269256 14729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 001f36501e0c44f090cf0e3ba3a05727: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 04:11:05.269693 14729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 001f36501e0c44f090cf0e3ba3a05727: Bootstrap complete.
I20260924 04:11:05.273896 14729 raft_consensus.cc:348] T 00000000000000000000000000000000 P 001f36501e0c44f090cf0e3ba3a05727 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "001f36501e0c44f090cf0e3ba3a05727" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36317 } }
I20260924 04:11:05.274343 14729 raft_consensus.cc:729] T 00000000000000000000000000000000 P 001f36501e0c44f090cf0e3ba3a05727 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 001f36501e0c44f090cf0e3ba3a05727, State: Initialized, Role: FOLLOWER
I20260924 04:11:05.274883 14729 consensus_queue.cc:260] T 00000000000000000000000000000000 P 001f36501e0c44f090cf0e3ba3a05727 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "001f36501e0c44f090cf0e3ba3a05727" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36317 } }
I20260924 04:11:05.275002 14729 raft_consensus.cc:388] T 00000000000000000000000000000000 P 001f36501e0c44f090cf0e3ba3a05727 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:11:05.275092 14729 raft_consensus.cc:482] T 00000000000000000000000000000000 P 001f36501e0c44f090cf0e3ba3a05727 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:11:05.275213 14729 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 001f36501e0c44f090cf0e3ba3a05727 [term 1 FOLLOWER]: Advancing to term 2
I20260924 04:11:05.277174 14729 raft_consensus.cc:504] T 00000000000000000000000000000000 P 001f36501e0c44f090cf0e3ba3a05727 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "001f36501e0c44f090cf0e3ba3a05727" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36317 } }
I20260924 04:11:05.277518 14729 leader_election.cc:302] T 00000000000000000000000000000000 P 001f36501e0c44f090cf0e3ba3a05727 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 001f36501e0c44f090cf0e3ba3a05727; no voters:
I20260924 04:11:05.277859 14729 leader_election.cc:288] T 00000000000000000000000000000000 P 001f36501e0c44f090cf0e3ba3a05727 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 04:11:05.277951 14733 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 001f36501e0c44f090cf0e3ba3a05727 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 04:11:05.278174 14733 raft_consensus.cc:686] T 00000000000000000000000000000000 P 001f36501e0c44f090cf0e3ba3a05727 [term 2 LEADER]: Becoming Leader. State: Replica: 001f36501e0c44f090cf0e3ba3a05727, State: Running, Role: LEADER
I20260924 04:11:05.278461 14733 consensus_queue.cc:237] T 00000000000000000000000000000000 P 001f36501e0c44f090cf0e3ba3a05727 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "001f36501e0c44f090cf0e3ba3a05727" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36317 } }
I20260924 04:11:05.279138 14729 sys_catalog.cc:564] T 00000000000000000000000000000000 P 001f36501e0c44f090cf0e3ba3a05727 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:11:05.279389 14733 sys_catalog.cc:455] T 00000000000000000000000000000000 P 001f36501e0c44f090cf0e3ba3a05727 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "001f36501e0c44f090cf0e3ba3a05727" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "001f36501e0c44f090cf0e3ba3a05727" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36317 } } }
I20260924 04:11:05.279496 14733 sys_catalog.cc:458] T 00000000000000000000000000000000 P 001f36501e0c44f090cf0e3ba3a05727 [sys.catalog]: This master's current role is: LEADER
I20260924 04:11:05.279878 14734 sys_catalog.cc:455] T 00000000000000000000000000000000 P 001f36501e0c44f090cf0e3ba3a05727 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 001f36501e0c44f090cf0e3ba3a05727. Latest consensus state: current_term: 2 leader_uuid: "001f36501e0c44f090cf0e3ba3a05727" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "001f36501e0c44f090cf0e3ba3a05727" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36317 } } }
I20260924 04:11:05.280031 14734 sys_catalog.cc:458] T 00000000000000000000000000000000 P 001f36501e0c44f090cf0e3ba3a05727 [sys.catalog]: This master's current role is: LEADER
I20260924 04:11:05.281034 14737 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:11:05.286057 14737 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:11:05.286643 14737 catalog_manager.cc:1259] Loaded cluster ID: 34b384b4f485405d98a779ee1e519a62
I20260924 04:11:05.286703 14737 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:11:05.288126 14737 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260924 04:11:05.292582 14737 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260924 04:11:05.297848 14737 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:11:05.298589 14737 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 001f36501e0c44f090cf0e3ba3a05727: Loaded TSK: 0
I20260924 04:11:05.299782 14737 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 04:11:05.368562 14731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:11:05.368892 14731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:11:05.368980 14731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:11:05.373101 14731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:11:05.373160 14731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:11:05.373246 14731 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:11:05.378799 14731 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:37167
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=34783
--tserver_master_addrs=127.0.98.62:36317
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14731
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:11:05.379827 14731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:11:05.380972 14731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:11:05.391494 14756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:11:05.391814 14757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:11:05.391494 14759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:11:05.392119 14731 server_base.cc:1034] running on GCE node
I20260924 04:11:05.392575 14731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:11:05.393395 14731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:11:05.394599 14731 hybrid_clock.cc:648] HybridClock initialized: now 1790223065394561 us; error 46 us; skew 500 ppm
Sep 24 04:11:05 dist-test-slave-00ld krb5kdc[14114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223065, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:11:05.397868 14731 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:11:05.398800 14731 webserver.cc:460] Webserver started at http://127.0.98.1:34783/ using document root <none> and password file <none>
I20260924 04:11:05.399376 14731 fs_manager.cc:359] Metadata directory not provided
I20260924 04:11:05.399451 14731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:11:05.403370 14731 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.000s
I20260924 04:11:05.405544 14766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:11:05.406812 14731 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:11:05.406962 14731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-0/wal
uuid: "a808f529633040fa88cd3449dd7017dc"
format_stamp: "Formatted at 2026-09-24 04:11:04 on dist-test-slave-00ld"
server_key: "c802f9bf344a5ec4b2682c6437d62edd"
server_key_iv: "1e3005a93af7b3af26bb524be11f2c12"
server_key_version: "encryptionkey@0"
I20260924 04:11:05.407490 14731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:11:05.419626 14731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:11:05.429508 14731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:11:05.429733 14731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:11:05.430333 14731 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:11:05.431437 14773 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 04:11:05.435487 14731 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 04:11:05.435577 14731 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260924 04:11:05.435679 14731 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 04:11:05.437389 14731 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 04:11:05.437459 14731 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260924 04:11:05.437774 14773 tablet_bootstrap.cc:492] T 7a824eb115b64f23a7afae2d6a3a1d3f P a808f529633040fa88cd3449dd7017dc: Bootstrap starting.
I20260924 04:11:05.447497 14731 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37167
I20260924 04:11:05.447602 14880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37167 every 8 connection(s)
I20260924 04:11:05.448460 14731 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:11:05.457953 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 14731
I20260924 04:11:05.457959 14773 log.cc:824] T 7a824eb115b64f23a7afae2d6a3a1d3f P a808f529633040fa88cd3449dd7017dc: Log is configured to *not* fsync() on all Append() calls
I20260924 04:11:05.458451 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:34039
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:36317
--webserver_port=37241
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 04:11:05 dist-test-slave-00ld krb5kdc[14114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223065, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:11:05.466574 14884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:05.450730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36317 (local address 127.0.98.1:52659)
0924 04:11:05.451373 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0924 04:11:05.451440 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0924 04:11:05.452529 (+ 1089us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:11:05.453680 (+ 1151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:11:05.453689 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:11:05.454257 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:11:05.454969 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:11:05.454989 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:11:05.456254 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:11:05.456258 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:11:05.457523 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:11:05.457537 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:11:05.458545 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:11:05.458552 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:11:05.458762 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:11:05.459451 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:11:05.459471 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:11:05.461397 (+ 1926us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:11:05.463465 (+ 2068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:11:05.463470 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:11:05.463482 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:11:05.463750 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:11:05.464123 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:11:05.464126 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:11:05.464128 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:11:05.464262 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:11:05.464880 (+ 618us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:11:05.464885 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:11:05.465143 (+ 258us) client_negotiation.cc:769] Sending connection context
0924 04:11:05.465323 (+ 180us) client_negotiation.cc:240] Negotiation successful
0924 04:11:05.465665 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":137,"threads_started":1}
I20260924 04:11:05.466558 14883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:05.450739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52659 (local address 127.0.98.62:36317)
0924 04:11:05.451155 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0924 04:11:05.451174 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 04:11:05.451597 (+ 423us) server_negotiation.cc:407] Connection header received
0924 04:11:05.452974 (+ 1377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:11:05.452985 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:11:05.453282 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:11:05.453541 (+ 259us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:11:05.455135 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:05.456094 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:05.457648 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:05.458392 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:05.458567 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:11:05.461594 (+ 3027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:11:05.461599 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:11:05.461611 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:11:05.461637 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:11:05.463338 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:11:05.463883 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:11:05.463888 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:11:05.463892 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:11:05.464009 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:11:05.464368 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:11:05.464376 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:11:05.464377 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:11:05.464769 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:11:05.464909 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 04:11:05.465447 (+ 538us) server_negotiation.cc:299] Negotiation successful
0924 04:11:05.465700 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":135,"threads_started":1}
I20260924 04:11:05.467571 14773 tablet_bootstrap.cc:492] T 7a824eb115b64f23a7afae2d6a3a1d3f P a808f529633040fa88cd3449dd7017dc: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 04:11:05.468036 14773 tablet_bootstrap.cc:492] T 7a824eb115b64f23a7afae2d6a3a1d3f P a808f529633040fa88cd3449dd7017dc: Bootstrap complete.
I20260924 04:11:05.468787 14881 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36317
I20260924 04:11:05.469040 14881 heartbeater.cc:464] Registering TS with master...
I20260924 04:11:05.469518 14881 heartbeater.cc:511] Master 127.0.98.62:36317 requested a full tablet report, sending...
I20260924 04:11:05.469600 14773 ts_tablet_manager.cc:1397] T 7a824eb115b64f23a7afae2d6a3a1d3f P a808f529633040fa88cd3449dd7017dc: Time spent bootstrapping tablet: real 0.032s user 0.015s sys 0.015s
I20260924 04:11:05.472124 14693 ts_manager.cc:194] Registered new tserver with Master: a808f529633040fa88cd3449dd7017dc (127.0.98.1:37167)
I20260924 04:11:05.472934 14773 raft_consensus.cc:348] T 7a824eb115b64f23a7afae2d6a3a1d3f P a808f529633040fa88cd3449dd7017dc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a808f529633040fa88cd3449dd7017dc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37167 } } peers { permanent_uuid: "c68b0618084a48a6aa96560ea5cf20f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45681 } } peers { permanent_uuid: "b57c26125c704ebf960138690cb649c3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34039 } }
I20260924 04:11:05.473466 14773 raft_consensus.cc:729] T 7a824eb115b64f23a7afae2d6a3a1d3f P a808f529633040fa88cd3449dd7017dc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a808f529633040fa88cd3449dd7017dc, State: Initialized, Role: FOLLOWER
I20260924 04:11:05.474025 14773 consensus_queue.cc:260] T 7a824eb115b64f23a7afae2d6a3a1d3f P a808f529633040fa88cd3449dd7017dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a808f529633040fa88cd3449dd7017dc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37167 } } peers { permanent_uuid: "c68b0618084a48a6aa96560ea5cf20f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45681 } } peers { permanent_uuid: "b57c26125c704ebf960138690cb649c3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34039 } }
I20260924 04:11:05.475019 14773 ts_tablet_manager.cc:1428] T 7a824eb115b64f23a7afae2d6a3a1d3f P a808f529633040fa88cd3449dd7017dc: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260924 04:11:05.476001 14693 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52659
I20260924 04:11:05.478507 14881 heartbeater.cc:503] Master 127.0.98.62:36317 was elected leader, sending a full tablet report...
Sep 24 04:11:05 dist-test-slave-00ld krb5kdc[14114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223065, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 04:11:05.486282 14889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:05.475398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58548 (local address 127.0.98.1:37167)
0924 04:11:05.475744 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0924 04:11:05.475755 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 04:11:05.476127 (+ 372us) server_negotiation.cc:407] Connection header received
0924 04:11:05.476388 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:11:05.476394 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:11:05.476581 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:11:05.476756 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:11:05.477448 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:05.478459 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:05.479781 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:05.480365 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:05.480466 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:11:05.482779 (+ 2313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:11:05.482786 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:11:05.482796 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:11:05.482816 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:11:05.484447 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:11:05.485006 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:11:05.485011 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:11:05.485012 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:11:05.485097 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:11:05.485373 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:11:05.485375 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:11:05.485376 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:11:05.485589 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:11:05.485685 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 04:11:05.486033 (+ 348us) server_negotiation.cc:299] Negotiation successful
0924 04:11:05.486165 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":97,"threads_started":1}
I20260924 04:11:05.486289 14888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:05.475335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37167 (local address 127.0.0.1:58548)
0924 04:11:05.475996 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0924 04:11:05.476045 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 04:11:05.476225 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:11:05.476758 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:11:05.476764 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:11:05.477183 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:11:05.477309 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:11:05.477317 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:11:05.478586 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:11:05.478589 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:11:05.479628 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:11:05.479638 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:11:05.480451 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:11:05.480463 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:11:05.480546 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:11:05.481147 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:11:05.481159 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:11:05.482636 (+ 1477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:11:05.484559 (+ 1923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:11:05.484568 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:11:05.484573 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:11:05.484893 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:11:05.485185 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:11:05.485188 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:11:05.485190 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:11:05.485282 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:11:05.485693 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:11:05.485698 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:11:05.485816 (+ 118us) client_negotiation.cc:769] Sending connection context
0924 04:11:05.485950 (+ 134us) client_negotiation.cc:240] Negotiation successful
0924 04:11:05.486126 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":104,"threads_started":1}
I20260924 04:11:05.487327 14815 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a824eb115b64f23a7afae2d6a3a1d3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:11:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58548
I20260924 04:11:05.488240 14890 tablet_replica.cc:330] stopping tablet replica
I20260924 04:11:05.488513 14890 raft_consensus.cc:2229] T 7a824eb115b64f23a7afae2d6a3a1d3f P a808f529633040fa88cd3449dd7017dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:11:05.488807 14890 raft_consensus.cc:2258] T 7a824eb115b64f23a7afae2d6a3a1d3f P a808f529633040fa88cd3449dd7017dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:11:05.489746 14890 ts_tablet_manager.cc:1905] T 7a824eb115b64f23a7afae2d6a3a1d3f P a808f529633040fa88cd3449dd7017dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:11:05.493252 14890 ts_tablet_manager.cc:1918] T 7a824eb115b64f23a7afae2d6a3a1d3f P a808f529633040fa88cd3449dd7017dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:11:05.493360 14890 log.cc:1196] T 7a824eb115b64f23a7afae2d6a3a1d3f P a808f529633040fa88cd3449dd7017dc: Deleting WAL directory at /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-0/wal/wals/7a824eb115b64f23a7afae2d6a3a1d3f
I20260924 04:11:05.493630 14890 ts_tablet_manager.cc:1939] T 7a824eb115b64f23a7afae2d6a3a1d3f P a808f529633040fa88cd3449dd7017dc: Deleting consensus metadata
I20260924 04:11:05.495164 14680 catalog_manager.cc:4841] TS a808f529633040fa88cd3449dd7017dc (127.0.98.1:37167): tablet 7a824eb115b64f23a7afae2d6a3a1d3f (table test-table [id=e6bd0a45223b4ebc8ce2adf8c797018d]) successfully deleted
W20260924 04:11:05.577733 14886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:11:05.578022 14886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:11:05.578105 14886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:11:05.582366 14886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:11:05.582427 14886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:11:05.582523 14886 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:11:05.587720 14886 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:34039
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=37241
--tserver_master_addrs=127.0.98.62:36317
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14886
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:11:05.588882 14886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:11:05.589957 14886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:11:05.600046 14896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:11:05.600051 14895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:11:05.600073 14898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:11:05.600051 14886 server_base.cc:1034] running on GCE node
I20260924 04:11:05.600872 14886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:11:05.601505 14886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:11:05.602684 14886 hybrid_clock.cc:648] HybridClock initialized: now 1790223065602663 us; error 33 us; skew 500 ppm
Sep 24 04:11:05 dist-test-slave-00ld krb5kdc[14114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223065, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:11:05.605731 14886 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:11:05.606628 14886 webserver.cc:460] Webserver started at http://127.0.98.2:37241/ using document root <none> and password file <none>
I20260924 04:11:05.607225 14886 fs_manager.cc:359] Metadata directory not provided
I20260924 04:11:05.607336 14886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:11:05.611642 14886 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 04:11:05.613957 14905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:11:05.615579 14886 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 04:11:05.615682 14886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-1/wal
uuid: "b57c26125c704ebf960138690cb649c3"
format_stamp: "Formatted at 2026-09-24 04:11:04 on dist-test-slave-00ld"
server_key: "7e9bf5bf9dc9dd220b3f395f38ae55ae"
server_key_iv: "c7bcbd016f811948b6457e1b3de06e63"
server_key_version: "encryptionkey@0"
I20260924 04:11:05.616241 14886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:11:05.629096 14886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:11:05.632143 14886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:11:05.632299 14886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:11:05.632917 14886 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:11:05.633711 14886 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:11:05.633759 14886 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:11:05.633797 14886 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:11:05.633813 14886 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:11:05.642482 14886 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34039
I20260924 04:11:05.642606 15018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34039 every 8 connection(s)
I20260924 04:11:05.643543 14886 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:11:05.645596 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 14886
I20260924 04:11:05.646067 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:45681
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:36317
--webserver_port=41653
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 04:11:05 dist-test-slave-00ld krb5kdc[14114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223065, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:11:05.659413 14883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:05.645406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54707 (local address 127.0.98.62:36317)
0924 04:11:05.645575 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 04:11:05.645579 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:11:05.646883 (+ 1304us) server_negotiation.cc:407] Connection header received
0924 04:11:05.648105 (+ 1222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:11:05.648108 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:11:05.648160 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:11:05.648236 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:11:05.649574 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:05.650091 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:05.651945 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:05.652170 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:05.652255 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:11:05.655202 (+ 2947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:11:05.655209 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:11:05.655214 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:11:05.655251 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:11:05.657268 (+ 2017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:11:05.657767 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:11:05.657774 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:11:05.657778 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:11:05.657880 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:11:05.658218 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:11:05.658225 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:11:05.658228 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:11:05.658506 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:11:05.658622 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 04:11:05.659104 (+ 482us) server_negotiation.cc:299] Negotiation successful
0924 04:11:05.659241 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260924 04:11:05.660215 15021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:05.645951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36317 (local address 127.0.98.2:54707)
0924 04:11:05.646723 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0924 04:11:05.646763 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 04:11:05.647853 (+ 1090us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:11:05.648472 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:11:05.648483 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:11:05.648973 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:11:05.649437 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:11:05.649450 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:11:05.650225 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:11:05.650232 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:11:05.651807 (+ 1575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:11:05.651822 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:11:05.652342 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:11:05.652349 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:11:05.652615 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:11:05.653280 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:11:05.653309 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:11:05.655022 (+ 1713us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:11:05.657390 (+ 2368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:11:05.657395 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:11:05.657408 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:11:05.657664 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:11:05.657990 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:11:05.657993 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:11:05.657994 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:11:05.658124 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:11:05.658619 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:11:05.658624 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:11:05.658865 (+ 241us) client_negotiation.cc:769] Sending connection context
0924 04:11:05.659033 (+ 168us) client_negotiation.cc:240] Negotiation successful
0924 04:11:05.659292 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":159,"threads_started":1}
I20260924 04:11:05.661552 15019 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36317
I20260924 04:11:05.661795 15019 heartbeater.cc:464] Registering TS with master...
I20260924 04:11:05.662295 15019 heartbeater.cc:511] Master 127.0.98.62:36317 requested a full tablet report, sending...
I20260924 04:11:05.663427 14693 ts_manager.cc:194] Registered new tserver with Master: b57c26125c704ebf960138690cb649c3 (127.0.98.2:34039)
I20260924 04:11:05.664077 14693 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:54707
W20260924 04:11:05.759444 15022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:11:05.759697 15022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:11:05.759740 15022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:11:05.763849 15022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:11:05.763904 15022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:11:05.764010 15022 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:11:05.769162 15022 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:45681
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=41653
--tserver_master_addrs=127.0.98.62:36317
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15022
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:11:05.770153 15022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:11:05.771157 15022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:11:05.780655 15028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:11:05.780871 15022 server_base.cc:1034] running on GCE node
W20260924 04:11:05.780656 15030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:11:05.780666 15027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:11:05.781582 15022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:11:05.782244 15022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:11:05.783427 15022 hybrid_clock.cc:648] HybridClock initialized: now 1790223065783418 us; error 29 us; skew 500 ppm
Sep 24 04:11:05 dist-test-slave-00ld krb5kdc[14114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223065, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:11:05.786055 15022 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:11:05.786984 15022 webserver.cc:460] Webserver started at http://127.0.98.3:41653/ using document root <none> and password file <none>
I20260924 04:11:05.787525 15022 fs_manager.cc:359] Metadata directory not provided
I20260924 04:11:05.787613 15022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:11:05.791287 15022 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:11:05.793309 15037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:11:05.794493 15022 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:11:05.794606 15022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-2/wal
uuid: "c68b0618084a48a6aa96560ea5cf20f7"
format_stamp: "Formatted at 2026-09-24 04:11:04 on dist-test-slave-00ld"
server_key: "d12377f81f2dd5b855781d00330550c9"
server_key_iv: "3575e29b7a2156dad63ff1b610294213"
server_key_version: "encryptionkey@0"
I20260924 04:11:05.795110 15022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:11:05.809183 15022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:11:05.812546 15022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:11:05.812793 15022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:11:05.813405 15022 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:11:05.814373 15022 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:11:05.814448 15022 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:11:05.814522 15022 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:11:05.814574 15022 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:11:05.823110 15022 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45681
I20260924 04:11:05.823159 15150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45681 every 8 connection(s)
I20260924 04:11:05.824184 15022 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 04:11:05 dist-test-slave-00ld krb5kdc[14114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223065, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:11:05.833910 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 15022
I20260924 04:11:05.838192 14883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:05.825990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34619 (local address 127.0.98.62:36317)
0924 04:11:05.826128 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 04:11:05.826132 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:11:05.827164 (+ 1032us) server_negotiation.cc:407] Connection header received
0924 04:11:05.828055 (+ 891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:11:05.828059 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:11:05.828116 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:11:05.828197 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:11:05.829488 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:05.830020 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:05.831236 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:05.831460 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:05.831595 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:11:05.834156 (+ 2561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:11:05.834160 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:11:05.834162 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:11:05.834187 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:11:05.836097 (+ 1910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:11:05.836619 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:11:05.836622 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:11:05.836624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:11:05.836744 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:11:05.837069 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:11:05.837073 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:11:05.837075 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:11:05.837263 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:11:05.837375 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 04:11:05.837891 (+ 516us) server_negotiation.cc:299] Negotiation successful
0924 04:11:05.838028 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 04:11:05.839207 15153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:05.826363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36317 (local address 127.0.98.3:34619)
0924 04:11:05.827005 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0924 04:11:05.827054 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 04:11:05.827847 (+ 793us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:11:05.828374 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:11:05.828381 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:11:05.828823 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:11:05.829333 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:11:05.829348 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:11:05.830161 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:11:05.830169 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:11:05.831094 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:11:05.831104 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:11:05.831654 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:11:05.831658 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:11:05.831829 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:11:05.832419 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:11:05.832436 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:11:05.833940 (+ 1504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:11:05.836220 (+ 2280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:11:05.836226 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:11:05.836240 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:11:05.836518 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:11:05.836821 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:11:05.836823 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:11:05.836824 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:11:05.836960 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:11:05.837372 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:11:05.837380 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:11:05.837651 (+ 271us) client_negotiation.cc:769] Sending connection context
0924 04:11:05.837850 (+ 199us) client_negotiation.cc:240] Negotiation successful
0924 04:11:05.838195 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":110,"threads_started":1}
I20260924 04:11:05.840548 15151 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36317
I20260924 04:11:05.840824 15151 heartbeater.cc:464] Registering TS with master...
I20260924 04:11:05.841394 15151 heartbeater.cc:511] Master 127.0.98.62:36317 requested a full tablet report, sending...
I20260924 04:11:05.842705 14693 ts_manager.cc:194] Registered new tserver with Master: c68b0618084a48a6aa96560ea5cf20f7 (127.0.98.3:45681)
I20260924 04:11:05.842998 15155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:05.834630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56340 (local address 127.0.98.62:36317)
0924 04:11:05.834904 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0924 04:11:05.834908 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:11:05.835085 (+ 177us) server_negotiation.cc:407] Connection header received
0924 04:11:05.835211 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:11:05.835216 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:11:05.835274 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:11:05.835389 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:11:05.836263 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:05.836945 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:05.837792 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:05.838030 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:05.838141 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:11:05.839542 (+ 1401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:11:05.839546 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:11:05.839548 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:11:05.839573 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:11:05.841248 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:11:05.841731 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:11:05.841734 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:11:05.841735 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:11:05.841806 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:11:05.842106 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:11:05.842109 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:11:05.842111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:11:05.842349 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:11:05.842544 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0924 04:11:05.842715 (+ 171us) server_negotiation.cc:299] Negotiation successful
0924 04:11:05.842840 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":74,"threads_started":1}
I20260924 04:11:05.843396 14693 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34619
I20260924 04:11:05.844835 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:11:05.849390 15155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:05.845494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56354 (local address 127.0.98.62:36317)
0924 04:11:05.845618 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 04:11:05.845623 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:11:05.845829 (+ 206us) server_negotiation.cc:407] Connection header received
0924 04:11:05.846022 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:11:05.846025 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:11:05.846128 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:11:05.846237 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:11:05.846481 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:05.846995 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:05.847858 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:05.848101 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:05.848213 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:11:05.848541 (+ 328us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:11:05.848996 (+ 455us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:11:05.849127 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0924 04:11:05.849167 (+ 40us) server_negotiation.cc:299] Negotiation successful
0924 04:11:05.849224 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 04:11:05.850047 14693 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:56354:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:11:05.853354 14693 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:11:05.861443 14815 tablet_service.cc:1467] Processing CreateTablet for tablet 020341c20950455d98af1603009a598c (DEFAULT_TABLE table=test-table [id=7160df39c8214273a3611730448e8c90]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:11:05.862149 14815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 020341c20950455d98af1603009a598c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:11:05.865375 14888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:05.860031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34039 (local address 127.0.0.1:46168)
0924 04:11:05.860165 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 04:11:05.860180 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:11:05.860259 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:11:05.861256 (+ 997us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:11:05.861259 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:11:05.861277 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:11:05.861380 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:11:05.861385 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:11:05.862983 (+ 1598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:11:05.862988 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:11:05.863852 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:11:05.863861 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:11:05.864923 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:11:05.864927 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:11:05.865022 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:11:05.865080 (+ 58us) client_negotiation.cc:769] Sending connection context
0924 04:11:05.865133 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 04:11:05.865219 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":1}
I20260924 04:11:05.865567 14773 tablet_bootstrap.cc:492] T 020341c20950455d98af1603009a598c P a808f529633040fa88cd3449dd7017dc: Bootstrap starting.
I20260924 04:11:05.866104 15161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:05.860453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45681 (local address 127.0.0.1:58910)
0924 04:11:05.860785 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0924 04:11:05.860802 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 04:11:05.860896 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:11:05.861606 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:11:05.861608 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:11:05.861626 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:11:05.861716 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:11:05.861722 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:11:05.863042 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:11:05.863045 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:11:05.864168 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:11:05.864178 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:11:05.865712 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:11:05.865716 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:11:05.865801 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:11:05.865822 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 04:11:05.865899 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 04:11:05.865972 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"mutex_wait_us":29,"thread_start_us":134,"threads_started":1}
I20260924 04:11:05.866200 15162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:05.860659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58910 (local address 127.0.98.3:45681)
0924 04:11:05.861138 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0924 04:11:05.861146 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 04:11:05.861163 (+ 17us) server_negotiation.cc:407] Connection header received
0924 04:11:05.861252 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:11:05.861257 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:11:05.861457 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:11:05.861626 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:11:05.861852 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:05.862896 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:05.864302 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:05.865566 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:05.865693 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:11:05.865847 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0924 04:11:05.865953 (+ 106us) server_negotiation.cc:299] Negotiation successful
0924 04:11:05.866053 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":158,"threads_started":1}
I20260924 04:11:05.866390 14773 tablet_bootstrap.cc:654] T 020341c20950455d98af1603009a598c P a808f529633040fa88cd3449dd7017dc: Neither blocks nor log segments found. Creating new log.
I20260924 04:11:05.867730 15160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:05.860350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46168 (local address 127.0.98.2:34039)
0924 04:11:05.860724 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0924 04:11:05.860737 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 04:11:05.860762 (+ 25us) server_negotiation.cc:407] Connection header received
0924 04:11:05.860844 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:11:05.860850 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:11:05.861101 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:11:05.861268 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:11:05.861489 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:05.862832 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:05.863983 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:05.864781 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:05.867273 (+ 2492us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:11:05.867389 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 04:11:05.867432 (+ 43us) server_negotiation.cc:299] Negotiation successful
0924 04:11:05.867577 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":131,"threads_started":1}
I20260924 04:11:05.867902 14773 tablet_bootstrap.cc:492] T 020341c20950455d98af1603009a598c P a808f529633040fa88cd3449dd7017dc: No bootstrap required, opened a new log
I20260924 04:11:05.867993 14773 ts_tablet_manager.cc:1397] T 020341c20950455d98af1603009a598c P a808f529633040fa88cd3449dd7017dc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260924 04:11:05.868461 14773 raft_consensus.cc:348] T 020341c20950455d98af1603009a598c P a808f529633040fa88cd3449dd7017dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b57c26125c704ebf960138690cb649c3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34039 } } peers { permanent_uuid: "c68b0618084a48a6aa96560ea5cf20f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45681 } } peers { permanent_uuid: "a808f529633040fa88cd3449dd7017dc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37167 } }
I20260924 04:11:05.868603 14773 raft_consensus.cc:374] T 020341c20950455d98af1603009a598c P a808f529633040fa88cd3449dd7017dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:11:05.868649 14773 raft_consensus.cc:729] T 020341c20950455d98af1603009a598c P a808f529633040fa88cd3449dd7017dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a808f529633040fa88cd3449dd7017dc, State: Initialized, Role: FOLLOWER
I20260924 04:11:05.868845 14773 consensus_queue.cc:260] T 020341c20950455d98af1603009a598c P a808f529633040fa88cd3449dd7017dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b57c26125c704ebf960138690cb649c3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34039 } } peers { permanent_uuid: "c68b0618084a48a6aa96560ea5cf20f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45681 } } peers { permanent_uuid: "a808f529633040fa88cd3449dd7017dc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37167 } }
I20260924 04:11:05.869589 14773 ts_tablet_manager.cc:1428] T 020341c20950455d98af1603009a598c P a808f529633040fa88cd3449dd7017dc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 04:11:05.870755 14953 tablet_service.cc:1467] Processing CreateTablet for tablet 020341c20950455d98af1603009a598c (DEFAULT_TABLE table=test-table [id=7160df39c8214273a3611730448e8c90]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:11:05.870834 15085 tablet_service.cc:1467] Processing CreateTablet for tablet 020341c20950455d98af1603009a598c (DEFAULT_TABLE table=test-table [id=7160df39c8214273a3611730448e8c90]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:11:05.871767 14953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 020341c20950455d98af1603009a598c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:11:05.871881 15085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 020341c20950455d98af1603009a598c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:11:05.878566 15165 tablet_bootstrap.cc:492] T 020341c20950455d98af1603009a598c P c68b0618084a48a6aa96560ea5cf20f7: Bootstrap starting.
I20260924 04:11:05.879027 15164 tablet_bootstrap.cc:492] T 020341c20950455d98af1603009a598c P b57c26125c704ebf960138690cb649c3: Bootstrap starting.
I20260924 04:11:05.880671 15165 tablet_bootstrap.cc:654] T 020341c20950455d98af1603009a598c P c68b0618084a48a6aa96560ea5cf20f7: Neither blocks nor log segments found. Creating new log.
I20260924 04:11:05.881290 15164 tablet_bootstrap.cc:654] T 020341c20950455d98af1603009a598c P b57c26125c704ebf960138690cb649c3: Neither blocks nor log segments found. Creating new log.
I20260924 04:11:05.881893 15165 log.cc:824] T 020341c20950455d98af1603009a598c P c68b0618084a48a6aa96560ea5cf20f7: Log is configured to *not* fsync() on all Append() calls
I20260924 04:11:05.882120 15164 log.cc:824] T 020341c20950455d98af1603009a598c P b57c26125c704ebf960138690cb649c3: Log is configured to *not* fsync() on all Append() calls
I20260924 04:11:05.884099 15165 tablet_bootstrap.cc:492] T 020341c20950455d98af1603009a598c P c68b0618084a48a6aa96560ea5cf20f7: No bootstrap required, opened a new log
I20260924 04:11:05.884099 15164 tablet_bootstrap.cc:492] T 020341c20950455d98af1603009a598c P b57c26125c704ebf960138690cb649c3: No bootstrap required, opened a new log
I20260924 04:11:05.884294 15164 ts_tablet_manager.cc:1397] T 020341c20950455d98af1603009a598c P b57c26125c704ebf960138690cb649c3: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 04:11:05.884294 15165 ts_tablet_manager.cc:1397] T 020341c20950455d98af1603009a598c P c68b0618084a48a6aa96560ea5cf20f7: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 04:11:05.888288 15165 raft_consensus.cc:348] T 020341c20950455d98af1603009a598c P c68b0618084a48a6aa96560ea5cf20f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b57c26125c704ebf960138690cb649c3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34039 } } peers { permanent_uuid: "c68b0618084a48a6aa96560ea5cf20f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45681 } } peers { permanent_uuid: "a808f529633040fa88cd3449dd7017dc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37167 } }
I20260924 04:11:05.888288 15164 raft_consensus.cc:348] T 020341c20950455d98af1603009a598c P b57c26125c704ebf960138690cb649c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b57c26125c704ebf960138690cb649c3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34039 } } peers { permanent_uuid: "c68b0618084a48a6aa96560ea5cf20f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45681 } } peers { permanent_uuid: "a808f529633040fa88cd3449dd7017dc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37167 } }
I20260924 04:11:05.888585 15164 raft_consensus.cc:374] T 020341c20950455d98af1603009a598c P b57c26125c704ebf960138690cb649c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:11:05.888585 15165 raft_consensus.cc:374] T 020341c20950455d98af1603009a598c P c68b0618084a48a6aa96560ea5cf20f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:11:05.888651 15164 raft_consensus.cc:729] T 020341c20950455d98af1603009a598c P b57c26125c704ebf960138690cb649c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b57c26125c704ebf960138690cb649c3, State: Initialized, Role: FOLLOWER
I20260924 04:11:05.888660 15165 raft_consensus.cc:729] T 020341c20950455d98af1603009a598c P c68b0618084a48a6aa96560ea5cf20f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c68b0618084a48a6aa96560ea5cf20f7, State: Initialized, Role: FOLLOWER
I20260924 04:11:05.889158 15165 consensus_queue.cc:260] T 020341c20950455d98af1603009a598c P c68b0618084a48a6aa96560ea5cf20f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b57c26125c704ebf960138690cb649c3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34039 } } peers { permanent_uuid: "c68b0618084a48a6aa96560ea5cf20f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45681 } } peers { permanent_uuid: "a808f529633040fa88cd3449dd7017dc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37167 } }
I20260924 04:11:05.889158 15164 consensus_queue.cc:260] T 020341c20950455d98af1603009a598c P b57c26125c704ebf960138690cb649c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b57c26125c704ebf960138690cb649c3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34039 } } peers { permanent_uuid: "c68b0618084a48a6aa96560ea5cf20f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45681 } } peers { permanent_uuid: "a808f529633040fa88cd3449dd7017dc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37167 } }
I20260924 04:11:05.889827 15019 heartbeater.cc:503] Master 127.0.98.62:36317 was elected leader, sending a full tablet report...
I20260924 04:11:05.890190 15164 ts_tablet_manager.cc:1428] T 020341c20950455d98af1603009a598c P b57c26125c704ebf960138690cb649c3: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260924 04:11:05.890190 15165 ts_tablet_manager.cc:1428] T 020341c20950455d98af1603009a598c P c68b0618084a48a6aa96560ea5cf20f7: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260924 04:11:05.890400 15151 heartbeater.cc:503] Master 127.0.98.62:36317 was elected leader, sending a full tablet report...
W20260924 04:11:05.894452 15020 tablet.cc:2369] T 020341c20950455d98af1603009a598c P b57c26125c704ebf960138690cb649c3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:11:05.931479 14887 raft_consensus.cc:482] T 020341c20950455d98af1603009a598c P a808f529633040fa88cd3449dd7017dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:11:05.931715 14887 raft_consensus.cc:504] T 020341c20950455d98af1603009a598c P a808f529633040fa88cd3449dd7017dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b57c26125c704ebf960138690cb649c3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34039 } } peers { permanent_uuid: "c68b0618084a48a6aa96560ea5cf20f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45681 } } peers { permanent_uuid: "a808f529633040fa88cd3449dd7017dc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37167 } }
I20260924 04:11:05.932933 14887 leader_election.cc:288] T 020341c20950455d98af1603009a598c P a808f529633040fa88cd3449dd7017dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b57c26125c704ebf960138690cb649c3 (127.0.98.2:34039), c68b0618084a48a6aa96560ea5cf20f7 (127.0.98.3:45681)
I20260924 04:11:05.937903 14884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:05.933098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34039 (local address 127.0.98.1:59091)
0924 04:11:05.933289 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0924 04:11:05.933306 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 04:11:05.933407 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:11:05.933784 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:11:05.933786 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:11:05.933799 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:11:05.933927 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:11:05.933933 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:11:05.935029 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:11:05.935031 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:11:05.936388 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:11:05.936400 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:11:05.937378 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:11:05.937381 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:11:05.937503 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:11:05.937581 (+ 78us) client_negotiation.cc:769] Sending connection context
0924 04:11:05.937656 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 04:11:05.937736 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"mutex_wait_us":68}
I20260924 04:11:05.937918 15160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:05.933239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59091 (local address 127.0.98.2:34039)
0924 04:11:05.933374 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 04:11:05.933379 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:11:05.933393 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:11:05.933592 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:11:05.933595 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:11:05.933670 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:11:05.933764 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:11:05.934049 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:05.934918 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:05.936541 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:05.937243 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:05.937364 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:11:05.937439 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 04:11:05.937713 (+ 274us) server_negotiation.cc:299] Negotiation successful
0924 04:11:05.937792 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 04:11:05.937918 15162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:05.933361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34809 (local address 127.0.98.3:45681)
0924 04:11:05.933534 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0924 04:11:05.933540 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:11:05.933677 (+ 137us) server_negotiation.cc:407] Connection header received
0924 04:11:05.933767 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:11:05.933770 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:11:05.933833 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:11:05.933916 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:11:05.934192 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:05.935034 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:05.936541 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:05.937243 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:05.937428 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:11:05.937498 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 04:11:05.937719 (+ 221us) server_negotiation.cc:299] Negotiation successful
0924 04:11:05.937792 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 04:11:05.937937 15170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:05.933215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45681 (local address 127.0.98.1:34809)
0924 04:11:05.933581 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0924 04:11:05.933591 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 04:11:05.933674 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:11:05.933938 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:11:05.933941 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:11:05.933956 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:11:05.934082 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:11:05.934087 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:11:05.935137 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:11:05.935140 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:11:05.936388 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:11:05.936400 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:11:05.937455 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:11:05.937459 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:11:05.937537 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:11:05.937573 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 04:11:05.937656 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 04:11:05.937736 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"mutex_wait_us":36,"thread_start_us":97,"threads_started":1}
I20260924 04:11:05.938625 14973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "020341c20950455d98af1603009a598c" candidate_uuid: "a808f529633040fa88cd3449dd7017dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b57c26125c704ebf960138690cb649c3" is_pre_election: true
I20260924 04:11:05.938748 15105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "020341c20950455d98af1603009a598c" candidate_uuid: "a808f529633040fa88cd3449dd7017dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c68b0618084a48a6aa96560ea5cf20f7" is_pre_election: true
I20260924 04:11:05.938952 14973 raft_consensus.cc:2454] T 020341c20950455d98af1603009a598c P b57c26125c704ebf960138690cb649c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a808f529633040fa88cd3449dd7017dc in term 0.
I20260924 04:11:05.938982 15105 raft_consensus.cc:2454] T 020341c20950455d98af1603009a598c P c68b0618084a48a6aa96560ea5cf20f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a808f529633040fa88cd3449dd7017dc in term 0.
I20260924 04:11:05.939498 14770 leader_election.cc:302] T 020341c20950455d98af1603009a598c P a808f529633040fa88cd3449dd7017dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: a808f529633040fa88cd3449dd7017dc, b57c26125c704ebf960138690cb649c3, c68b0618084a48a6aa96560ea5cf20f7; no voters:
I20260924 04:11:05.939803 14887 raft_consensus.cc:2781] T 020341c20950455d98af1603009a598c P a808f529633040fa88cd3449dd7017dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:11:05.939903 14887 raft_consensus.cc:482] T 020341c20950455d98af1603009a598c P a808f529633040fa88cd3449dd7017dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:11:05.939993 14887 raft_consensus.cc:3037] T 020341c20950455d98af1603009a598c P a808f529633040fa88cd3449dd7017dc [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:11:05.941130 14887 raft_consensus.cc:504] T 020341c20950455d98af1603009a598c P a808f529633040fa88cd3449dd7017dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b57c26125c704ebf960138690cb649c3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34039 } } peers { permanent_uuid: "c68b0618084a48a6aa96560ea5cf20f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45681 } } peers { permanent_uuid: "a808f529633040fa88cd3449dd7017dc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37167 } }
I20260924 04:11:05.941473 14887 leader_election.cc:288] T 020341c20950455d98af1603009a598c P a808f529633040fa88cd3449dd7017dc [CANDIDATE]: Term 1 election: Requested vote from peers b57c26125c704ebf960138690cb649c3 (127.0.98.2:34039), c68b0618084a48a6aa96560ea5cf20f7 (127.0.98.3:45681)
I20260924 04:11:05.941874 14973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "020341c20950455d98af1603009a598c" candidate_uuid: "a808f529633040fa88cd3449dd7017dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b57c26125c704ebf960138690cb649c3"
I20260924 04:11:05.941879 15105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "020341c20950455d98af1603009a598c" candidate_uuid: "a808f529633040fa88cd3449dd7017dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c68b0618084a48a6aa96560ea5cf20f7"
I20260924 04:11:05.942022 14973 raft_consensus.cc:3037] T 020341c20950455d98af1603009a598c P b57c26125c704ebf960138690cb649c3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:11:05.942039 15105 raft_consensus.cc:3037] T 020341c20950455d98af1603009a598c P c68b0618084a48a6aa96560ea5cf20f7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:11:05.943190 14973 raft_consensus.cc:2454] T 020341c20950455d98af1603009a598c P b57c26125c704ebf960138690cb649c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a808f529633040fa88cd3449dd7017dc in term 1.
I20260924 04:11:05.943289 15105 raft_consensus.cc:2454] T 020341c20950455d98af1603009a598c P c68b0618084a48a6aa96560ea5cf20f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a808f529633040fa88cd3449dd7017dc in term 1.
I20260924 04:11:05.943507 14767 leader_election.cc:302] T 020341c20950455d98af1603009a598c P a808f529633040fa88cd3449dd7017dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a808f529633040fa88cd3449dd7017dc, b57c26125c704ebf960138690cb649c3; no voters:
I20260924 04:11:05.943701 14887 raft_consensus.cc:2781] T 020341c20950455d98af1603009a598c P a808f529633040fa88cd3449dd7017dc [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:11:05.943943 14887 raft_consensus.cc:686] T 020341c20950455d98af1603009a598c P a808f529633040fa88cd3449dd7017dc [term 1 LEADER]: Becoming Leader. State: Replica: a808f529633040fa88cd3449dd7017dc, State: Running, Role: LEADER
I20260924 04:11:05.944243 14887 consensus_queue.cc:237] T 020341c20950455d98af1603009a598c P a808f529633040fa88cd3449dd7017dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b57c26125c704ebf960138690cb649c3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34039 } } peers { permanent_uuid: "c68b0618084a48a6aa96560ea5cf20f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45681 } } peers { permanent_uuid: "a808f529633040fa88cd3449dd7017dc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37167 } }
I20260924 04:11:05.947114 14692 catalog_manager.cc:5495] T 020341c20950455d98af1603009a598c P a808f529633040fa88cd3449dd7017dc reported cstate change: term changed from 0 to 1, leader changed from <none> to a808f529633040fa88cd3449dd7017dc (127.0.98.1). New cstate: current_term: 1 leader_uuid: "a808f529633040fa88cd3449dd7017dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b57c26125c704ebf960138690cb649c3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c68b0618084a48a6aa96560ea5cf20f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a808f529633040fa88cd3449dd7017dc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37167 } health_report { overall_health: HEALTHY } } }
W20260924 04:11:05.949712 14882 tablet.cc:2369] T 020341c20950455d98af1603009a598c P a808f529633040fa88cd3449dd7017dc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:11:05.971879 14889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:05.968008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58560 (local address 127.0.98.1:37167)
0924 04:11:05.968144 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 04:11:05.968148 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:11:05.968165 (+ 17us) server_negotiation.cc:407] Connection header received
0924 04:11:05.968249 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:11:05.968251 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:11:05.968307 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:11:05.968415 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:11:05.968871 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:05.969474 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:05.970342 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:05.970603 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:05.970696 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:11:05.971005 (+ 309us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:11:05.971486 (+ 481us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:11:05.971585 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 04:11:05.971689 (+ 104us) server_negotiation.cc:299] Negotiation successful
0924 04:11:05.971747 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 04:11:05.975975 15105 raft_consensus.cc:1261] T 020341c20950455d98af1603009a598c P c68b0618084a48a6aa96560ea5cf20f7 [term 1 FOLLOWER]: Refusing update from remote peer a808f529633040fa88cd3449dd7017dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:11:05.975975 14973 raft_consensus.cc:1261] T 020341c20950455d98af1603009a598c P b57c26125c704ebf960138690cb649c3 [term 1 FOLLOWER]: Refusing update from remote peer a808f529633040fa88cd3449dd7017dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:11:05.976765 14887 consensus_queue.cc:1035] T 020341c20950455d98af1603009a598c P a808f529633040fa88cd3449dd7017dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c68b0618084a48a6aa96560ea5cf20f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:11:05.976984 15171 consensus_queue.cc:1035] T 020341c20950455d98af1603009a598c P a808f529633040fa88cd3449dd7017dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b57c26125c704ebf960138690cb649c3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:11:05.984118 15174 mvcc.cc:205] Tried to move back new op lower bound from 7332753678231240704 to 7332753678110486528. Current Snapshot: MvccSnapshot[applied={T|T < 7332753678231240704}]
I20260924 04:11:05.985397 15175 mvcc.cc:205] Tried to move back new op lower bound from 7332753678231240704 to 7332753678110486528. Current Snapshot: MvccSnapshot[applied={T|T < 7332753678231240704}]
I20260924 04:11:05.992599 14693 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:56354:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:11:05.992810 14693 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:56354:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:11:05.995182 14693 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 001f36501e0c44f090cf0e3ba3a05727: Sending DeleteTablet for 3 replicas of tablet 020341c20950455d98af1603009a598c
I20260924 04:11:05.995879 14815 tablet_service.cc:1514] Processing DeleteTablet for tablet 020341c20950455d98af1603009a598c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:11:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58548
I20260924 04:11:05.995954 14953 tablet_service.cc:1514] Processing DeleteTablet for tablet 020341c20950455d98af1603009a598c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:11:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46168
I20260924 04:11:05.996101 15085 tablet_service.cc:1514] Processing DeleteTablet for tablet 020341c20950455d98af1603009a598c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:11:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58910
I20260924 04:11:05.996297 15184 tablet_replica.cc:330] T 020341c20950455d98af1603009a598c P a808f529633040fa88cd3449dd7017dc: stopping tablet replica
I20260924 04:11:05.996441 15184 raft_consensus.cc:2229] T 020341c20950455d98af1603009a598c P a808f529633040fa88cd3449dd7017dc [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:11:05.996598 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 14731
I20260924 04:11:05.996840 15186 tablet_replica.cc:330] T 020341c20950455d98af1603009a598c P c68b0618084a48a6aa96560ea5cf20f7: stopping tablet replica
I20260924 04:11:05.997022 15185 tablet_replica.cc:330] T 020341c20950455d98af1603009a598c P b57c26125c704ebf960138690cb649c3: stopping tablet replica
I20260924 04:11:05.997344 15186 raft_consensus.cc:2229] T 020341c20950455d98af1603009a598c P c68b0618084a48a6aa96560ea5cf20f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:11:05.997646 15186 raft_consensus.cc:2258] T 020341c20950455d98af1603009a598c P c68b0618084a48a6aa96560ea5cf20f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:11:05.998092 15185 raft_consensus.cc:2229] T 020341c20950455d98af1603009a598c P b57c26125c704ebf960138690cb649c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:11:05.998371 15185 raft_consensus.cc:2258] T 020341c20950455d98af1603009a598c P b57c26125c704ebf960138690cb649c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:11:05.999372 15186 ts_tablet_manager.cc:1905] T 020341c20950455d98af1603009a598c P c68b0618084a48a6aa96560ea5cf20f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:11:05.999843 15185 ts_tablet_manager.cc:1905] T 020341c20950455d98af1603009a598c P b57c26125c704ebf960138690cb649c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:11:06.001644 15186 ts_tablet_manager.cc:1918] T 020341c20950455d98af1603009a598c P c68b0618084a48a6aa96560ea5cf20f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:11:06.001765 15186 log.cc:1196] T 020341c20950455d98af1603009a598c P c68b0618084a48a6aa96560ea5cf20f7: Deleting WAL directory at /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-2/wal/wals/020341c20950455d98af1603009a598c
I20260924 04:11:06.002096 15186 ts_tablet_manager.cc:1939] T 020341c20950455d98af1603009a598c P c68b0618084a48a6aa96560ea5cf20f7: Deleting consensus metadata
I20260924 04:11:06.003118 14680 catalog_manager.cc:4841] TS c68b0618084a48a6aa96560ea5cf20f7 (127.0.98.3:45681): tablet 020341c20950455d98af1603009a598c (table test-table [id=7160df39c8214273a3611730448e8c90]) successfully deleted
I20260924 04:11:06.003196 15185 ts_tablet_manager.cc:1918] T 020341c20950455d98af1603009a598c P b57c26125c704ebf960138690cb649c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:11:06.003309 15185 log.cc:1196] T 020341c20950455d98af1603009a598c P b57c26125c704ebf960138690cb649c3: Deleting WAL directory at /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-1/wal/wals/020341c20950455d98af1603009a598c
I20260924 04:11:06.003669 15185 ts_tablet_manager.cc:1939] T 020341c20950455d98af1603009a598c P b57c26125c704ebf960138690cb649c3: Deleting consensus metadata
W20260924 04:11:06.004627 14680 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37167 (error 108)
I20260924 04:11:06.004877 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 14886
I20260924 04:11:06.004895 14677 catalog_manager.cc:4841] TS b57c26125c704ebf960138690cb649c3 (127.0.98.2:34039): tablet 020341c20950455d98af1603009a598c (table test-table [id=7160df39c8214273a3611730448e8c90]) successfully deleted
I20260924 04:11:06.007009 15161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:06.006664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37167 (local address 127.0.0.1:58564)
0924 04:11:06.006827 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 04:11:06.006907 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260924 04:11:06.007280 14680 catalog_manager.cc:4584] TS a808f529633040fa88cd3449dd7017dc (127.0.98.1:37167): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 020341c20950455d98af1603009a598c: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37167: connect: Connection refused (error 111)
I20260924 04:11:06.010613 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 15022
I20260924 04:11:06.017952 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 14660
I20260924 04:11:06.024591 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36317
--webserver_interface=127.0.98.62
--webserver_port=35995
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36317
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:11:06.137804 15188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:11:06.138131 15188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:11:06.138217 15188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:11:06.142323 15188 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:11:06.142411 15188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:11:06.142452 15188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:11:06.142513 15188 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:11:06.142553 15188 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:11:06.147823 15188 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36317
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=35995
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15188
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:11:06.148923 15188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:11:06.149931 15188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:11:06.158444 15196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:11:06.158502 15193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:11:06.158490 15194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:11:06.158500 15188 server_base.cc:1034] running on GCE node
I20260924 04:11:06.159276 15188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:11:06.160363 15188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:11:06.161562 15188 hybrid_clock.cc:648] HybridClock initialized: now 1790223066161532 us; error 42 us; skew 500 ppm
Sep 24 04:11:06 dist-test-slave-00ld krb5kdc[14114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223066, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:11:06.164182 15188 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:11:06.165194 15188 webserver.cc:460] Webserver started at http://127.0.98.62:35995/ using document root <none> and password file <none>
I20260924 04:11:06.165751 15188 fs_manager.cc:359] Metadata directory not provided
I20260924 04:11:06.165843 15188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:11:06.170006 15188 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:11:06.172236 15203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:11:06.173629 15188 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:11:06.173764 15188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/master-0/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/master-0/wal
uuid: "001f36501e0c44f090cf0e3ba3a05727"
format_stamp: "Formatted at 2026-09-24 04:11:04 on dist-test-slave-00ld"
server_key: "ad4519d5f762093b834ef33571f1a933"
server_key_iv: "7c2fc7657d5745075655eafc99715370"
server_key_version: "encryptionkey@0"
I20260924 04:11:06.174345 15188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:11:06.187839 15188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:11:06.190946 15188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:11:06.191157 15188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:11:06.198792 15188 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36317
I20260924 04:11:06.199105 15255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36317 every 8 connection(s)
I20260924 04:11:06.199971 15188 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:11:06.200721 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 15188
I20260924 04:11:06.201212 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:37167
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:36317
--webserver_port=34783
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:11:06.205307 15256 sys_catalog.cc:263] Verifying existing consensus state
I20260924 04:11:06.206913 15256 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 001f36501e0c44f090cf0e3ba3a05727: Bootstrap starting.
I20260924 04:11:06.225050 15256 log.cc:824] T 00000000000000000000000000000000 P 001f36501e0c44f090cf0e3ba3a05727: Log is configured to *not* fsync() on all Append() calls
I20260924 04:11:06.236867 15256 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 001f36501e0c44f090cf0e3ba3a05727: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 04:11:06.237314 15256 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 001f36501e0c44f090cf0e3ba3a05727: Bootstrap complete.
I20260924 04:11:06.241403 15256 raft_consensus.cc:348] T 00000000000000000000000000000000 P 001f36501e0c44f090cf0e3ba3a05727 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "001f36501e0c44f090cf0e3ba3a05727" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36317 } }
I20260924 04:11:06.241802 15256 raft_consensus.cc:729] T 00000000000000000000000000000000 P 001f36501e0c44f090cf0e3ba3a05727 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 001f36501e0c44f090cf0e3ba3a05727, State: Initialized, Role: FOLLOWER
I20260924 04:11:06.242340 15256 consensus_queue.cc:260] T 00000000000000000000000000000000 P 001f36501e0c44f090cf0e3ba3a05727 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "001f36501e0c44f090cf0e3ba3a05727" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36317 } }
I20260924 04:11:06.242460 15256 raft_consensus.cc:388] T 00000000000000000000000000000000 P 001f36501e0c44f090cf0e3ba3a05727 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:11:06.242560 15256 raft_consensus.cc:482] T 00000000000000000000000000000000 P 001f36501e0c44f090cf0e3ba3a05727 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:11:06.242687 15256 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 001f36501e0c44f090cf0e3ba3a05727 [term 2 FOLLOWER]: Advancing to term 3
I20260924 04:11:06.244670 15256 raft_consensus.cc:504] T 00000000000000000000000000000000 P 001f36501e0c44f090cf0e3ba3a05727 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "001f36501e0c44f090cf0e3ba3a05727" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36317 } }
I20260924 04:11:06.245092 15256 leader_election.cc:302] T 00000000000000000000000000000000 P 001f36501e0c44f090cf0e3ba3a05727 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 001f36501e0c44f090cf0e3ba3a05727; no voters:
I20260924 04:11:06.245462 15256 leader_election.cc:288] T 00000000000000000000000000000000 P 001f36501e0c44f090cf0e3ba3a05727 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 04:11:06.245571 15260 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 001f36501e0c44f090cf0e3ba3a05727 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 04:11:06.245765 15260 raft_consensus.cc:686] T 00000000000000000000000000000000 P 001f36501e0c44f090cf0e3ba3a05727 [term 3 LEADER]: Becoming Leader. State: Replica: 001f36501e0c44f090cf0e3ba3a05727, State: Running, Role: LEADER
I20260924 04:11:06.246075 15260 consensus_queue.cc:237] T 00000000000000000000000000000000 P 001f36501e0c44f090cf0e3ba3a05727 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "001f36501e0c44f090cf0e3ba3a05727" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36317 } }
I20260924 04:11:06.246758 15256 sys_catalog.cc:564] T 00000000000000000000000000000000 P 001f36501e0c44f090cf0e3ba3a05727 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:11:06.247040 15260 sys_catalog.cc:455] T 00000000000000000000000000000000 P 001f36501e0c44f090cf0e3ba3a05727 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 001f36501e0c44f090cf0e3ba3a05727. Latest consensus state: current_term: 3 leader_uuid: "001f36501e0c44f090cf0e3ba3a05727" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "001f36501e0c44f090cf0e3ba3a05727" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36317 } } }
I20260924 04:11:06.247162 15260 sys_catalog.cc:458] T 00000000000000000000000000000000 P 001f36501e0c44f090cf0e3ba3a05727 [sys.catalog]: This master's current role is: LEADER
I20260924 04:11:06.247515 15264 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:11:06.247599 15261 sys_catalog.cc:455] T 00000000000000000000000000000000 P 001f36501e0c44f090cf0e3ba3a05727 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "001f36501e0c44f090cf0e3ba3a05727" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "001f36501e0c44f090cf0e3ba3a05727" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36317 } } }
I20260924 04:11:06.247780 15261 sys_catalog.cc:458] T 00000000000000000000000000000000 P 001f36501e0c44f090cf0e3ba3a05727 [sys.catalog]: This master's current role is: LEADER
I20260924 04:11:06.253084 15264 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:11:06.253906 15264 catalog_manager.cc:1259] Loaded cluster ID: 34b384b4f485405d98a779ee1e519a62
I20260924 04:11:06.253988 15264 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260924 04:11:06.255087 15264 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f462a28b9ad google::LogMessage::Fail() at ??:0
@ 0x7f462a28e88a google::LogMessage::SendToLog() at ??:0
@ 0x7f462a28b4de google::LogMessage::Flush() at ??:0
@ 0x7f462a28ca19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f4634bcefdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f4634bcfd2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f4634bcc579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f4634c10640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f462c4dd3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f462c4ddc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f462c4df3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f462c4ce512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f462db126db start_thread at ??:0
@ 0x7f4628d8961f clone at ??:0
W20260924 04:11:06.342939 15258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:11:06.343454 15258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:11:06.343533 15258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:11:06.348297 15258 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:11:06.348363 15258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:11:06.348472 15258 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:11:06.356133 15258 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:37167
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=34783
--tserver_master_addrs=127.0.98.62:36317
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15258
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:11:06.357543 15258 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:11:06.359051 15258 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:11:06.371438 15300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:11:06.371464 15297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:11:06.371667 15258 server_base.cc:1034] running on GCE node
W20260924 04:11:06.371886 15298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:11:06.372357 15258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:11:06.373179 15258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:11:06.374333 15258 hybrid_clock.cc:648] HybridClock initialized: now 1790223066374301 us; error 50 us; skew 500 ppm
Sep 24 04:11:06 dist-test-slave-00ld krb5kdc[14114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223066, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:11:06.377202 15258 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:11:06.378139 15258 webserver.cc:460] Webserver started at http://127.0.98.1:34783/ using document root <none> and password file <none>
I20260924 04:11:06.378702 15258 fs_manager.cc:359] Metadata directory not provided
I20260924 04:11:06.378764 15258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:11:06.382856 15258 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:11:06.385281 15307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:11:06.387034 15258 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:11:06.387158 15258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-0/wal
uuid: "a808f529633040fa88cd3449dd7017dc"
format_stamp: "Formatted at 2026-09-24 04:11:04 on dist-test-slave-00ld"
server_key: "c802f9bf344a5ec4b2682c6437d62edd"
server_key_iv: "1e3005a93af7b3af26bb524be11f2c12"
server_key_version: "encryptionkey@0"
I20260924 04:11:06.387960 15258 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:11:06.406663 15258 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:11:06.409991 15258 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:11:06.410189 15258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:11:06.410780 15258 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:11:06.412032 15314 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 04:11:06.417440 15258 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 04:11:06.417557 15258 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260924 04:11:06.417658 15258 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 04:11:06.419332 15258 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 04:11:06.419409 15258 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260924 04:11:06.419703 15314 tablet_bootstrap.cc:492] T 020341c20950455d98af1603009a598c P a808f529633040fa88cd3449dd7017dc: Bootstrap starting.
I20260924 04:11:06.429971 15258 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37167
I20260924 04:11:06.430119 15421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37167 every 8 connection(s)
I20260924 04:11:06.430964 15258 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:11:06.438616 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 15258
I20260924 04:11:06.439076 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:34039
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:36317
--webserver_port=37241
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:11:06.441684 15314 log.cc:824] T 020341c20950455d98af1603009a598c P a808f529633040fa88cd3449dd7017dc: Log is configured to *not* fsync() on all Append() calls
I20260924 04:11:06.451037 15314 tablet_bootstrap.cc:492] T 020341c20950455d98af1603009a598c P a808f529633040fa88cd3449dd7017dc: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 04:11:06.451489 15314 tablet_bootstrap.cc:492] T 020341c20950455d98af1603009a598c P a808f529633040fa88cd3449dd7017dc: Bootstrap complete.
I20260924 04:11:06.452957 15314 ts_tablet_manager.cc:1397] T 020341c20950455d98af1603009a598c P a808f529633040fa88cd3449dd7017dc: Time spent bootstrapping tablet: real 0.033s user 0.026s sys 0.007s
I20260924 04:11:06.456315 15314 raft_consensus.cc:348] T 020341c20950455d98af1603009a598c P a808f529633040fa88cd3449dd7017dc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b57c26125c704ebf960138690cb649c3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34039 } } peers { permanent_uuid: "c68b0618084a48a6aa96560ea5cf20f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45681 } } peers { permanent_uuid: "a808f529633040fa88cd3449dd7017dc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37167 } }
I20260924 04:11:06.456893 15314 raft_consensus.cc:729] T 020341c20950455d98af1603009a598c P a808f529633040fa88cd3449dd7017dc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a808f529633040fa88cd3449dd7017dc, State: Initialized, Role: FOLLOWER
I20260924 04:11:06.457445 15314 consensus_queue.cc:260] T 020341c20950455d98af1603009a598c P a808f529633040fa88cd3449dd7017dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b57c26125c704ebf960138690cb649c3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34039 } } peers { permanent_uuid: "c68b0618084a48a6aa96560ea5cf20f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45681 } } peers { permanent_uuid: "a808f529633040fa88cd3449dd7017dc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37167 } }
I20260924 04:11:06.458559 15314 ts_tablet_manager.cc:1428] T 020341c20950455d98af1603009a598c P a808f529633040fa88cd3449dd7017dc: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260924 04:11:06.489296 15424 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:11:06.432896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36317 (local address 127.0.98.1:40199)
0924 04:11:06.433376 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0924 04:11:06.433429 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 04:11:06.434221 (+ 792us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:11:06.488435 (+ 54214us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36317: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":142,"threads_started":1}
W20260924 04:11:06.489804 15422 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36317 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:36317: Client connection negotiation failed: client connection to 127.0.98.62:36317: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 04:11:06.490563 15424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:06.490320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36317 (local address 127.0.98.1:45403)
0924 04:11:06.490442 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 04:11:06.490512 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260924 04:11:06.491240 15424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:06.491023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36317 (local address 127.0.98.1:53039)
0924 04:11:06.491153 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 04:11:06.491194 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260924 04:11:06.491468 15422 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 04:11:06.552877 15425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:11:06.553138 15425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:11:06.553182 15425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:11:06.557376 15425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:11:06.557436 15425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:11:06.557528 15425 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:11:06.562676 15425 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:34039
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=37241
--tserver_master_addrs=127.0.98.62:36317
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15425
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:11:06.563738 15425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:11:06.564813 15425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:11:06.574244 15435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:11:06.574290 15432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:11:06.574393 15433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:11:06.574465 15425 server_base.cc:1034] running on GCE node
I20260924 04:11:06.574934 15425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:11:06.575609 15425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:11:06.576756 15425 hybrid_clock.cc:648] HybridClock initialized: now 1790223066576733 us; error 34 us; skew 500 ppm
Sep 24 04:11:06 dist-test-slave-00ld krb5kdc[14114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223066, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:11:06.579419 15425 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:11:06.580263 15425 webserver.cc:460] Webserver started at http://127.0.98.2:37241/ using document root <none> and password file <none>
I20260924 04:11:06.580869 15425 fs_manager.cc:359] Metadata directory not provided
I20260924 04:11:06.580958 15425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:11:06.584556 15425 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260924 04:11:06.586654 15442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:11:06.587836 15425 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:11:06.587970 15425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-1/wal
uuid: "b57c26125c704ebf960138690cb649c3"
format_stamp: "Formatted at 2026-09-24 04:11:04 on dist-test-slave-00ld"
server_key: "7e9bf5bf9dc9dd220b3f395f38ae55ae"
server_key_iv: "c7bcbd016f811948b6457e1b3de06e63"
server_key_version: "encryptionkey@0"
I20260924 04:11:06.588531 15425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:11:06.602752 15425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:11:06.606321 15425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:11:06.606532 15425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:11:06.607151 15425 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:11:06.607985 15425 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:11:06.608062 15425 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:11:06.608130 15425 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:11:06.608181 15425 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:11:06.617127 15425 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34039
I20260924 04:11:06.617189 15555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34039 every 8 connection(s)
I20260924 04:11:06.618183 15425 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:11:06.621768 15558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:06.620300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36317 (local address 127.0.98.2:56177)
0924 04:11:06.620979 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0924 04:11:06.621093 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":100,"threads_started":1}
W20260924 04:11:06.622226 15556 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36317 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:36317: Client connection negotiation failed: client connection to 127.0.98.62:36317: connect: Connection refused (error 111)
I20260924 04:11:06.622843 15558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:06.622593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36317 (local address 127.0.98.2:48195)
0924 04:11:06.622739 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 04:11:06.622788 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260924 04:11:06.623481 15558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:06.623291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36317 (local address 127.0.98.2:35121)
0924 04:11:06.623385 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0924 04:11:06.623432 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260924 04:11:06.623675 15556 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 04:11:06.626027 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 15425
I20260924 04:11:06.626437 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:45681
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:36317
--webserver_port=41653
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:11:06.736634 15559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:11:06.736971 15559 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:11:06.737053 15559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:11:06.741075 15559 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:11:06.741174 15559 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:11:06.741269 15559 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:11:06.746469 15559 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:45681
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=41653
--tserver_master_addrs=127.0.98.62:36317
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15559
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:11:06.747582 15559 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:11:06.748667 15559 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:11:06.758617 15564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:11:06.758685 15565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:11:06.758672 15559 server_base.cc:1034] running on GCE node
W20260924 04:11:06.758635 15567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:11:06.759279 15559 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:11:06.759964 15559 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:11:06.761144 15559 hybrid_clock.cc:648] HybridClock initialized: now 1790223066761126 us; error 28 us; skew 500 ppm
Sep 24 04:11:06 dist-test-slave-00ld krb5kdc[14114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223066, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:11:06.763835 15559 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:11:06.764801 15559 webserver.cc:460] Webserver started at http://127.0.98.3:41653/ using document root <none> and password file <none>
I20260924 04:11:06.765362 15559 fs_manager.cc:359] Metadata directory not provided
I20260924 04:11:06.765455 15559 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:11:06.769375 15559 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:11:06.771610 15574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:11:06.773000 15559 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:11:06.773121 15559 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-2/wal
uuid: "c68b0618084a48a6aa96560ea5cf20f7"
format_stamp: "Formatted at 2026-09-24 04:11:04 on dist-test-slave-00ld"
server_key: "d12377f81f2dd5b855781d00330550c9"
server_key_iv: "3575e29b7a2156dad63ff1b610294213"
server_key_version: "encryptionkey@0"
I20260924 04:11:06.773645 15559 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:11:06.784941 15559 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:11:06.787741 15559 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:11:06.787938 15559 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:11:06.788509 15559 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:11:06.789371 15559 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:11:06.789443 15559 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:11:06.789511 15559 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:11:06.789568 15559 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:11:06.798494 15559 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45681
I20260924 04:11:06.798513 15687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45681 every 8 connection(s)
I20260924 04:11:06.799542 15559 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790222898807720-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:11:06.802569 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 15559
I20260924 04:11:06.803346 15690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:06.801652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36317 (local address 127.0.98.3:55825)
0924 04:11:06.802353 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0924 04:11:06.802508 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":111,"threads_started":1}
W20260924 04:11:06.803766 15688 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36317 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:36317: Client connection negotiation failed: client connection to 127.0.98.62:36317: connect: Connection refused (error 111)
I20260924 04:11:06.804306 15690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:06.804098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36317 (local address 127.0.98.3:43571)
0924 04:11:06.804205 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0924 04:11:06.804257 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260924 04:11:06.804889 15690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:06.804671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36317 (local address 127.0.98.3:59197)
0924 04:11:06.804792 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0924 04:11:06.804836 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260924 04:11:06.805049 15688 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 04:11:07.492496 15692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:07.492098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36317 (local address 127.0.98.1:41265)
0924 04:11:07.492333 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0924 04:11:07.492386 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":69,"threads_started":1}
I20260924 04:11:07.602936 15693 raft_consensus.cc:482] T 020341c20950455d98af1603009a598c P a808f529633040fa88cd3449dd7017dc [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:11:07.603170 15693 raft_consensus.cc:504] T 020341c20950455d98af1603009a598c P a808f529633040fa88cd3449dd7017dc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b57c26125c704ebf960138690cb649c3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34039 } } peers { permanent_uuid: "c68b0618084a48a6aa96560ea5cf20f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45681 } } peers { permanent_uuid: "a808f529633040fa88cd3449dd7017dc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37167 } }
I20260924 04:11:07.604278 15693 leader_election.cc:288] T 020341c20950455d98af1603009a598c P a808f529633040fa88cd3449dd7017dc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b57c26125c704ebf960138690cb649c3 (127.0.98.2:34039), c68b0618084a48a6aa96560ea5cf20f7 (127.0.98.3:45681)
Sep 24 04:11:07 dist-test-slave-00ld krb5kdc[14114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223066, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Sep 24 04:11:07 dist-test-slave-00ld krb5kdc[14114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223066, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260924 04:11:07.618584 15692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:07.604510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34039 (local address 127.0.98.1:57645)
0924 04:11:07.604622 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 04:11:07.604639 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 04:11:07.604802 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:11:07.606612 (+ 1810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:11:07.606617 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:11:07.607177 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:11:07.607695 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:11:07.607710 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:11:07.609050 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:11:07.609052 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:11:07.609988 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:11:07.609998 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:11:07.610704 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:11:07.610707 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:11:07.610886 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:11:07.611666 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:11:07.611677 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:11:07.613974 (+ 2297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:11:07.616292 (+ 2318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:11:07.616295 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:11:07.616297 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:11:07.616523 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:11:07.617024 (+ 501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:11:07.617027 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:11:07.617029 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:11:07.617117 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:11:07.617832 (+ 715us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:11:07.617837 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:11:07.618057 (+ 220us) client_negotiation.cc:769] Sending connection context
0924 04:11:07.618206 (+ 149us) client_negotiation.cc:240] Negotiation successful
0924 04:11:07.618449 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":150,"spinlock_wait_cycles":45824}
I20260924 04:11:07.618804 15696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:07.604826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57645 (local address 127.0.98.2:34039)
0924 04:11:07.605212 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0924 04:11:07.605227 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 04:11:07.605270 (+ 43us) server_negotiation.cc:407] Connection header received
0924 04:11:07.605889 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:11:07.605894 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:11:07.606236 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:11:07.606564 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:11:07.607882 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:07.608894 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:07.610109 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:07.610576 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:07.610760 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:11:07.614161 (+ 3401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:11:07.614166 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:11:07.614178 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:11:07.614205 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:11:07.616182 (+ 1977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:11:07.616627 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:11:07.616633 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:11:07.616641 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:11:07.616810 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:11:07.617218 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:11:07.617221 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:11:07.617222 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:11:07.617722 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:11:07.617852 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0924 04:11:07.618349 (+ 497us) server_negotiation.cc:299] Negotiation successful
0924 04:11:07.618632 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":80,"threads_started":1}
I20260924 04:11:07.620217 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:07.604517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45681 (local address 127.0.98.1:38953)
0924 04:11:07.605003 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0924 04:11:07.605019 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:11:07.605134 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:11:07.606611 (+ 1477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:11:07.606617 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:11:07.607177 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:11:07.607696 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:11:07.607710 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:11:07.609051 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:11:07.609054 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:11:07.609987 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:11:07.609997 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:11:07.610949 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:11:07.610952 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:11:07.611027 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:11:07.611636 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:11:07.611658 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:11:07.613547 (+ 1889us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:11:07.615759 (+ 2212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:11:07.615766 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:11:07.615780 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:11:07.616122 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:11:07.616598 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:11:07.616601 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:11:07.616603 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:11:07.616810 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:11:07.619731 (+ 2921us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:11:07.619735 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:11:07.619828 (+ 93us) client_negotiation.cc:769] Sending connection context
0924 04:11:07.619927 (+ 99us) client_negotiation.cc:240] Negotiation successful
0924 04:11:07.620059 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404,"mutex_wait_us":220,"spinlock_wait_cycles":4352,"thread_start_us":90,"threads_started":1}
I20260924 04:11:07.620563 15695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:07.604747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38953 (local address 127.0.98.3:45681)
0924 04:11:07.605068 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0924 04:11:07.605088 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 04:11:07.605139 (+ 51us) server_negotiation.cc:407] Connection header received
0924 04:11:07.605858 (+ 719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:11:07.605863 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:11:07.606236 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:11:07.606558 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:11:07.607873 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:07.608895 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:07.610141 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:07.610608 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:07.610759 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:11:07.613811 (+ 3052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:11:07.613816 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:11:07.613829 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:11:07.613858 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:11:07.615639 (+ 1781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:11:07.616263 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:11:07.616271 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:11:07.616278 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:11:07.616470 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:11:07.619167 (+ 2697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:11:07.619171 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:11:07.619175 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:11:07.619533 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:11:07.619676 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 04:11:07.620096 (+ 420us) server_negotiation.cc:299] Negotiation successful
0924 04:11:07.620375 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":111,"threads_started":1}
W20260924 04:11:07.621524 15308 leader_election.cc:341] T 020341c20950455d98af1603009a598c P a808f529633040fa88cd3449dd7017dc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b57c26125c704ebf960138690cb649c3 (127.0.98.2:34039): Not found: Tablet not found: 020341c20950455d98af1603009a598c
W20260924 04:11:07.622514 15311 leader_election.cc:341] T 020341c20950455d98af1603009a598c P a808f529633040fa88cd3449dd7017dc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c68b0618084a48a6aa96560ea5cf20f7 (127.0.98.3:45681): Not found: Tablet not found: 020341c20950455d98af1603009a598c
I20260924 04:11:07.622606 15311 leader_election.cc:302] T 020341c20950455d98af1603009a598c P a808f529633040fa88cd3449dd7017dc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a808f529633040fa88cd3449dd7017dc; no voters: b57c26125c704ebf960138690cb649c3, c68b0618084a48a6aa96560ea5cf20f7
I20260924 04:11:07.622877 15693 raft_consensus.cc:2726] T 020341c20950455d98af1603009a598c P a808f529633040fa88cd3449dd7017dc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 04:11:07.624577 15697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:07.624100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36317 (local address 127.0.98.2:43621)
0924 04:11:07.624437 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0924 04:11:07.624490 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":62,"threads_started":1}
I20260924 04:11:07.805913 15698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:07.805535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36317 (local address 127.0.98.3:33941)
0924 04:11:07.805770 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0924 04:11:07.805823 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":66,"threads_started":1}
I20260924 04:11:08.493832 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:08.493360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36317 (local address 127.0.98.1:57819)
0924 04:11:08.493625 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0924 04:11:08.493686 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":71,"threads_started":1}
I20260924 04:11:08.625628 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:08.625247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36317 (local address 127.0.98.2:53593)
0924 04:11:08.625498 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0924 04:11:08.625549 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":58,"threads_started":1}
I20260924 04:11:08.807045 15701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:08.806652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36317 (local address 127.0.98.3:37005)
0924 04:11:08.806893 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0924 04:11:08.806947 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":81,"threads_started":1}
I20260924 04:11:09.494973 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:09.494540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36317 (local address 127.0.98.1:45373)
0924 04:11:09.494799 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0924 04:11:09.494859 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":76,"threads_started":1}
I20260924 04:11:09.505072 15703 raft_consensus.cc:482] T 020341c20950455d98af1603009a598c P a808f529633040fa88cd3449dd7017dc [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:11:09.505189 15703 raft_consensus.cc:504] T 020341c20950455d98af1603009a598c P a808f529633040fa88cd3449dd7017dc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b57c26125c704ebf960138690cb649c3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34039 } } peers { permanent_uuid: "c68b0618084a48a6aa96560ea5cf20f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45681 } } peers { permanent_uuid: "a808f529633040fa88cd3449dd7017dc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37167 } }
I20260924 04:11:09.505578 15703 leader_election.cc:288] T 020341c20950455d98af1603009a598c P a808f529633040fa88cd3449dd7017dc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b57c26125c704ebf960138690cb649c3 (127.0.98.2:34039), c68b0618084a48a6aa96560ea5cf20f7 (127.0.98.3:45681)
W20260924 04:11:09.506381 15308 leader_election.cc:341] T 020341c20950455d98af1603009a598c P a808f529633040fa88cd3449dd7017dc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b57c26125c704ebf960138690cb649c3 (127.0.98.2:34039): Not found: Tablet not found: 020341c20950455d98af1603009a598c
W20260924 04:11:09.506533 15311 leader_election.cc:341] T 020341c20950455d98af1603009a598c P a808f529633040fa88cd3449dd7017dc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c68b0618084a48a6aa96560ea5cf20f7 (127.0.98.3:45681): Not found: Tablet not found: 020341c20950455d98af1603009a598c
I20260924 04:11:09.506595 15311 leader_election.cc:302] T 020341c20950455d98af1603009a598c P a808f529633040fa88cd3449dd7017dc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a808f529633040fa88cd3449dd7017dc; no voters: b57c26125c704ebf960138690cb649c3, c68b0618084a48a6aa96560ea5cf20f7
I20260924 04:11:09.506716 15703 raft_consensus.cc:2726] T 020341c20950455d98af1603009a598c P a808f529633040fa88cd3449dd7017dc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 04:11:09.626689 15704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:09.626296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36317 (local address 127.0.98.2:33993)
0924 04:11:09.626540 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0924 04:11:09.626592 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":78,"threads_started":1}
I20260924 04:11:09.808112 15705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:09.807733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36317 (local address 127.0.98.3:40775)
0924 04:11:09.807979 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0924 04:11:09.808032 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":83,"threads_started":1}
I20260924 04:11:10.496356 15706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:10.495841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36317 (local address 127.0.98.1:52643)
0924 04:11:10.496139 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0924 04:11:10.496211 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":84,"threads_started":1}
I20260924 04:11:10.627815 15707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:10.627431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36317 (local address 127.0.98.2:35413)
0924 04:11:10.627671 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0924 04:11:10.627725 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":76,"threads_started":1}
I20260924 04:11:10.809252 15708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:10.808846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36317 (local address 127.0.98.3:41911)
0924 04:11:10.809106 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0924 04:11:10.809161 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":85,"threads_started":1}
I20260924 04:11:11.497635 15709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:11.497217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36317 (local address 127.0.98.1:60615)
0924 04:11:11.497482 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0924 04:11:11.497536 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":76,"threads_started":1}
I20260924 04:11:11.629093 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:11.628549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36317 (local address 127.0.98.2:40161)
0924 04:11:11.628862 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0924 04:11:11.628949 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":62,"threads_started":1}
I20260924 04:11:11.810397 15711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:11.810003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36317 (local address 127.0.98.3:53983)
0924 04:11:11.810246 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0924 04:11:11.810302 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":81,"threads_started":1}
I20260924 04:11:11.922390 15712 raft_consensus.cc:482] T 020341c20950455d98af1603009a598c P a808f529633040fa88cd3449dd7017dc [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:11:11.922518 15712 raft_consensus.cc:504] T 020341c20950455d98af1603009a598c P a808f529633040fa88cd3449dd7017dc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b57c26125c704ebf960138690cb649c3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34039 } } peers { permanent_uuid: "c68b0618084a48a6aa96560ea5cf20f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45681 } } peers { permanent_uuid: "a808f529633040fa88cd3449dd7017dc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37167 } }
I20260924 04:11:11.922894 15712 leader_election.cc:288] T 020341c20950455d98af1603009a598c P a808f529633040fa88cd3449dd7017dc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b57c26125c704ebf960138690cb649c3 (127.0.98.2:34039), c68b0618084a48a6aa96560ea5cf20f7 (127.0.98.3:45681)
W20260924 04:11:11.923661 15308 leader_election.cc:341] T 020341c20950455d98af1603009a598c P a808f529633040fa88cd3449dd7017dc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b57c26125c704ebf960138690cb649c3 (127.0.98.2:34039): Not found: Tablet not found: 020341c20950455d98af1603009a598c
W20260924 04:11:11.923751 15311 leader_election.cc:341] T 020341c20950455d98af1603009a598c P a808f529633040fa88cd3449dd7017dc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c68b0618084a48a6aa96560ea5cf20f7 (127.0.98.3:45681): Not found: Tablet not found: 020341c20950455d98af1603009a598c
I20260924 04:11:11.923803 15311 leader_election.cc:302] T 020341c20950455d98af1603009a598c P a808f529633040fa88cd3449dd7017dc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a808f529633040fa88cd3449dd7017dc; no voters: b57c26125c704ebf960138690cb649c3, c68b0618084a48a6aa96560ea5cf20f7
I20260924 04:11:11.923945 15712 raft_consensus.cc:2726] T 020341c20950455d98af1603009a598c P a808f529633040fa88cd3449dd7017dc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 04:11:12.498862 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:12.498461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36317 (local address 127.0.98.1:58447)
0924 04:11:12.498710 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0924 04:11:12.498768 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":81,"threads_started":1}
I20260924 04:11:12.630329 15714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:12.629904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36317 (local address 127.0.98.2:38431)
0924 04:11:12.630179 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0924 04:11:12.630239 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":84,"threads_started":1}
I20260924 04:11:12.811558 15715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:12.811190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36317 (local address 127.0.98.3:36687)
0924 04:11:12.811417 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0924 04:11:12.811470 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":70,"threads_started":1}
I20260924 04:11:13.500017 15716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:13.499550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36317 (local address 127.0.98.1:54719)
0924 04:11:13.499850 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0924 04:11:13.499919 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":100,"threads_started":1}
I20260924 04:11:13.631645 15717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:13.631166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36317 (local address 127.0.98.2:55699)
0924 04:11:13.631453 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0924 04:11:13.631511 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":84,"threads_started":1}
I20260924 04:11:13.812902 15718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:13.812378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36317 (local address 127.0.98.3:33973)
0924 04:11:13.812740 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0924 04:11:13.812808 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":74,"threads_started":1}
I20260924 04:11:14.501247 15719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:14.500855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36317 (local address 127.0.98.1:55495)
0924 04:11:14.501094 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0924 04:11:14.501151 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":71,"threads_started":1}
I20260924 04:11:14.632905 15720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:14.632440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36317 (local address 127.0.98.2:34961)
0924 04:11:14.632751 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0924 04:11:14.632808 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":90,"threads_started":1}
I20260924 04:11:14.814208 15721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:14.813761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36317 (local address 127.0.98.3:55193)
0924 04:11:14.814048 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0924 04:11:14.814109 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":83,"threads_started":1}
I20260924 04:11:15.502396 15722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:15.501994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36317 (local address 127.0.98.1:40545)
0924 04:11:15.502243 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0924 04:11:15.502303 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":72,"threads_started":1}
I20260924 04:11:15.633991 15723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:15.633602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36317 (local address 127.0.98.2:42871)
0924 04:11:15.633849 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0924 04:11:15.633903 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":84,"threads_started":1}
I20260924 04:11:15.815420 15724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:15.814940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36317 (local address 127.0.98.3:43781)
0924 04:11:15.815250 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0924 04:11:15.815328 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":91,"threads_started":1}
I20260924 04:11:15.996258 15725 raft_consensus.cc:482] T 020341c20950455d98af1603009a598c P a808f529633040fa88cd3449dd7017dc [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:11:15.996402 15725 raft_consensus.cc:504] T 020341c20950455d98af1603009a598c P a808f529633040fa88cd3449dd7017dc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b57c26125c704ebf960138690cb649c3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34039 } } peers { permanent_uuid: "c68b0618084a48a6aa96560ea5cf20f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45681 } } peers { permanent_uuid: "a808f529633040fa88cd3449dd7017dc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37167 } }
I20260924 04:11:15.996847 15725 leader_election.cc:288] T 020341c20950455d98af1603009a598c P a808f529633040fa88cd3449dd7017dc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b57c26125c704ebf960138690cb649c3 (127.0.98.2:34039), c68b0618084a48a6aa96560ea5cf20f7 (127.0.98.3:45681)
W20260924 04:11:15.997656 15308 leader_election.cc:341] T 020341c20950455d98af1603009a598c P a808f529633040fa88cd3449dd7017dc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b57c26125c704ebf960138690cb649c3 (127.0.98.2:34039): Not found: Tablet not found: 020341c20950455d98af1603009a598c
W20260924 04:11:15.997756 15311 leader_election.cc:341] T 020341c20950455d98af1603009a598c P a808f529633040fa88cd3449dd7017dc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c68b0618084a48a6aa96560ea5cf20f7 (127.0.98.3:45681): Not found: Tablet not found: 020341c20950455d98af1603009a598c
I20260924 04:11:15.997813 15311 leader_election.cc:302] T 020341c20950455d98af1603009a598c P a808f529633040fa88cd3449dd7017dc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a808f529633040fa88cd3449dd7017dc; no voters: b57c26125c704ebf960138690cb649c3, c68b0618084a48a6aa96560ea5cf20f7
I20260924 04:11:15.997959 15725 raft_consensus.cc:2726] T 020341c20950455d98af1603009a598c P a808f529633040fa88cd3449dd7017dc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 04:11:16.503741 15726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:16.503334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36317 (local address 127.0.98.1:52935)
0924 04:11:16.503587 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0924 04:11:16.503635 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":108,"threads_started":1}
I20260924 04:11:16.635221 15727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:16.634813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36317 (local address 127.0.98.2:57823)
0924 04:11:16.635076 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0924 04:11:16.635127 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":99,"threads_started":1}
I20260924 04:11:16.816808 15728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:16.816291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36317 (local address 127.0.98.3:38767)
0924 04:11:16.816576 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0924 04:11:16.816635 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":76,"threads_started":1}
I20260924 04:11:17.504972 15729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:17.504501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36317 (local address 127.0.98.1:59087)
0924 04:11:17.504827 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0924 04:11:17.504881 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":87,"threads_started":1}
I20260924 04:11:17.636618 15730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:17.635962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36317 (local address 127.0.98.2:55979)
0924 04:11:17.636316 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0924 04:11:17.636485 (+ 169us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":97,"threads_started":1}
I20260924 04:11:17.818123 15731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:17.817702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36317 (local address 127.0.98.3:60791)
0924 04:11:17.817964 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0924 04:11:17.818026 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":88,"threads_started":1}
I20260924 04:11:18.506291 15732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:18.505822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36317 (local address 127.0.98.1:34823)
0924 04:11:18.506120 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0924 04:11:18.506187 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":92,"threads_started":1}
I20260924 04:11:18.638083 15733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:18.637578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36317 (local address 127.0.98.2:57089)
0924 04:11:18.637862 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0924 04:11:18.637935 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":80,"threads_started":1}
I20260924 04:11:18.819437 15734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:18.818947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36317 (local address 127.0.98.3:56997)
0924 04:11:18.819281 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0924 04:11:18.819340 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":122,"threads_started":1}
I20260924 04:11:19.507512 15735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:19.507094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36317 (local address 127.0.98.1:42173)
0924 04:11:19.507351 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0924 04:11:19.507413 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":73,"threads_started":1}
I20260924 04:11:19.639299 15736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:19.638900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36317 (local address 127.0.98.2:53837)
0924 04:11:19.639142 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0924 04:11:19.639200 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":77,"threads_started":1}
I20260924 04:11:19.820633 15737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:19.820220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36317 (local address 127.0.98.3:38201)
0924 04:11:19.820468 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0924 04:11:19.820529 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":83,"threads_started":1}
I20260924 04:11:20.508857 15738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:20.508403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36317 (local address 127.0.98.1:34785)
0924 04:11:20.508643 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0924 04:11:20.508753 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":76,"threads_started":1}
I20260924 04:11:20.640733 15739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:20.640238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36317 (local address 127.0.98.2:40233)
0924 04:11:20.640503 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0924 04:11:20.640576 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":78,"threads_started":1}
I20260924 04:11:20.821870 15740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:20.821436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36317 (local address 127.0.98.3:52045)
0924 04:11:20.821715 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0924 04:11:20.821776 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":83,"threads_started":1}
I20260924 04:11:21.373929 15741 raft_consensus.cc:482] T 020341c20950455d98af1603009a598c P a808f529633040fa88cd3449dd7017dc [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:11:21.374107 15741 raft_consensus.cc:504] T 020341c20950455d98af1603009a598c P a808f529633040fa88cd3449dd7017dc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b57c26125c704ebf960138690cb649c3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34039 } } peers { permanent_uuid: "c68b0618084a48a6aa96560ea5cf20f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45681 } } peers { permanent_uuid: "a808f529633040fa88cd3449dd7017dc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37167 } }
I20260924 04:11:21.374630 15741 leader_election.cc:288] T 020341c20950455d98af1603009a598c P a808f529633040fa88cd3449dd7017dc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b57c26125c704ebf960138690cb649c3 (127.0.98.2:34039), c68b0618084a48a6aa96560ea5cf20f7 (127.0.98.3:45681)
W20260924 04:11:21.375379 15311 leader_election.cc:341] T 020341c20950455d98af1603009a598c P a808f529633040fa88cd3449dd7017dc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c68b0618084a48a6aa96560ea5cf20f7 (127.0.98.3:45681): Not found: Tablet not found: 020341c20950455d98af1603009a598c
W20260924 04:11:21.375466 15308 leader_election.cc:341] T 020341c20950455d98af1603009a598c P a808f529633040fa88cd3449dd7017dc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b57c26125c704ebf960138690cb649c3 (127.0.98.2:34039): Not found: Tablet not found: 020341c20950455d98af1603009a598c
I20260924 04:11:21.375662 15308 leader_election.cc:302] T 020341c20950455d98af1603009a598c P a808f529633040fa88cd3449dd7017dc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a808f529633040fa88cd3449dd7017dc; no voters: b57c26125c704ebf960138690cb649c3, c68b0618084a48a6aa96560ea5cf20f7
I20260924 04:11:21.375852 15741 raft_consensus.cc:2726] T 020341c20950455d98af1603009a598c P a808f529633040fa88cd3449dd7017dc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 04:11:21.510295 15742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:21.509854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36317 (local address 127.0.98.1:40621)
0924 04:11:21.510128 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0924 04:11:21.510191 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":117,"threads_started":1}
I20260924 04:11:21.642323 15743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:21.641809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36317 (local address 127.0.98.2:57179)
0924 04:11:21.642152 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0924 04:11:21.642213 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":87,"threads_started":1}
I20260924 04:11:21.807371 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 15258
I20260924 04:11:21.813323 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 15425
I20260924 04:11:21.818979 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 15559
2026-09-24T04:11:21Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19026 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:11:21 dist-test-slave-00ld krb5kdc[15746](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:11:21 dist-test-slave-00ld krb5kdc[15746](info): set up 2 sockets
Sep 24 04:11:21 dist-test-slave-00ld krb5kdc[15746](info): commencing operation
krb5kdc: starting...
W20260924 04:11:22.857007 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/krb5kdc/test-user.keytab.
Sep 24 04:11:22 dist-test-slave-00ld krb5kdc[15746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:11:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:11:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:11:22.980041 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37025
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37025
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:11:23.090247 15759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:11:23.090557 15759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:11:23.090628 15759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:11:23.094681 15759 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 04:11:23.094764 15759 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 04:11:23.094790 15759 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 04:11:23.094815 15759 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:11:23.094856 15759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:11:23.094900 15759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:11:23.094933 15759 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:11:23.094966 15759 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:11:23.100162 15759 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37025
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15759
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:11:23.101212 15759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:11:23.102238 15759 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:11:23.110816 15765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:11:23.110849 15759 server_base.cc:1034] running on GCE node
W20260924 04:11:23.110816 15767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:11:23.110857 15764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:11:23.111575 15759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:11:23.112672 15759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:11:23.113884 15759 hybrid_clock.cc:648] HybridClock initialized: now 1790223083113873 us; error 23 us; skew 500 ppm
Sep 24 04:11:23 dist-test-slave-00ld krb5kdc[15746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223083, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:11:23.116451 15759 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:11:23.117322 15759 webserver.cc:460] Webserver started at http://127.0.98.62:37799/ using document root <none> and password file <none>
I20260924 04:11:23.117856 15759 fs_manager.cc:359] Metadata directory not provided
I20260924 04:11:23.117944 15759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:11:23.118165 15759 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:11:23.120090 15759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/master-0/data/instance:
uuid: "4b8cbc143cac4cd8bc3e990221c4f28e"
format_stamp: "Formatted at 2026-09-24 04:11:23 on dist-test-slave-00ld"
server_key: "e6e20e1cb40c8368cc2c1055d97fae4e"
server_key_iv: "a1aaffcaf6851dba7712560290aec965"
server_key_version: "encryptionkey@0"
I20260924 04:11:23.120648 15759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/master-0/wal/instance:
uuid: "4b8cbc143cac4cd8bc3e990221c4f28e"
format_stamp: "Formatted at 2026-09-24 04:11:23 on dist-test-slave-00ld"
server_key: "e6e20e1cb40c8368cc2c1055d97fae4e"
server_key_iv: "a1aaffcaf6851dba7712560290aec965"
server_key_version: "encryptionkey@0"
I20260924 04:11:23.125051 15759 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 04:11:23.127441 15774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:11:23.128767 15759 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:11:23.128897 15759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/master-0/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/master-0/wal
uuid: "4b8cbc143cac4cd8bc3e990221c4f28e"
format_stamp: "Formatted at 2026-09-24 04:11:23 on dist-test-slave-00ld"
server_key: "e6e20e1cb40c8368cc2c1055d97fae4e"
server_key_iv: "a1aaffcaf6851dba7712560290aec965"
server_key_version: "encryptionkey@0"
I20260924 04:11:23.129006 15759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:11:23.141119 15759 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:11:23.144343 15759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:11:23.144567 15759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:11:23.151899 15759 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37025
I20260924 04:11:23.152187 15826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37025 every 8 connection(s)
I20260924 04:11:23.153000 15759 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:11:23.155696 15827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:11:23.156059 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 15759
I20260924 04:11:23.156201 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/master-0/wal/instance
I20260924 04:11:23.156503 392 external_mini_cluster.cc:1428] Setting key ccc824369e26a942e6063a7ff3558464
I20260924 04:11:23.162801 15827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b8cbc143cac4cd8bc3e990221c4f28e: Bootstrap starting.
I20260924 04:11:23.165578 15827 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b8cbc143cac4cd8bc3e990221c4f28e: Neither blocks nor log segments found. Creating new log.
Sep 24 04:11:23 dist-test-slave-00ld krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:11:23.166497 15827 log.cc:824] T 00000000000000000000000000000000 P 4b8cbc143cac4cd8bc3e990221c4f28e: Log is configured to *not* fsync() on all Append() calls
I20260924 04:11:23.168473 15827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b8cbc143cac4cd8bc3e990221c4f28e: No bootstrap required, opened a new log
I20260924 04:11:23.170866 15830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:23.157919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59992 (local address 127.0.98.62:37025)
0924 04:11:23.158340 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0924 04:11:23.158360 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 04:11:23.158407 (+ 47us) server_negotiation.cc:407] Connection header received
0924 04:11:23.159073 (+ 666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:11:23.159098 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:11:23.159432 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:11:23.159706 (+ 274us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:11:23.160756 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:23.161727 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:23.162866 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:23.163389 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:23.163502 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:11:23.166280 (+ 2778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:11:23.166285 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:11:23.166298 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:11:23.166325 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:11:23.168037 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:11:23.168508 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:11:23.168513 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:11:23.168517 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:11:23.168672 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:11:23.169014 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:11:23.169016 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:11:23.169018 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:11:23.169406 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:11:23.169574 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0924 04:11:23.169849 (+ 275us) server_negotiation.cc:299] Negotiation successful
0924 04:11:23.170161 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":147,"threads_started":1}
I20260924 04:11:23.172377 15827 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b8cbc143cac4cd8bc3e990221c4f28e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b8cbc143cac4cd8bc3e990221c4f28e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37025 } }
I20260924 04:11:23.172609 15827 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b8cbc143cac4cd8bc3e990221c4f28e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:11:23.172701 15827 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b8cbc143cac4cd8bc3e990221c4f28e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b8cbc143cac4cd8bc3e990221c4f28e, State: Initialized, Role: FOLLOWER
I20260924 04:11:23.173166 15827 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b8cbc143cac4cd8bc3e990221c4f28e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b8cbc143cac4cd8bc3e990221c4f28e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37025 } }
I20260924 04:11:23.173293 15827 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b8cbc143cac4cd8bc3e990221c4f28e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:11:23.173342 15827 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b8cbc143cac4cd8bc3e990221c4f28e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:11:23.173410 15827 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b8cbc143cac4cd8bc3e990221c4f28e [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:11:23.174221 15827 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b8cbc143cac4cd8bc3e990221c4f28e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b8cbc143cac4cd8bc3e990221c4f28e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37025 } }
I20260924 04:11:23.174557 15827 leader_election.cc:302] T 00000000000000000000000000000000 P 4b8cbc143cac4cd8bc3e990221c4f28e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b8cbc143cac4cd8bc3e990221c4f28e; no voters:
I20260924 04:11:23.174860 15827 leader_election.cc:288] T 00000000000000000000000000000000 P 4b8cbc143cac4cd8bc3e990221c4f28e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:11:23.174998 15832 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b8cbc143cac4cd8bc3e990221c4f28e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:11:23.175240 15832 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b8cbc143cac4cd8bc3e990221c4f28e [term 1 LEADER]: Becoming Leader. State: Replica: 4b8cbc143cac4cd8bc3e990221c4f28e, State: Running, Role: LEADER
I20260924 04:11:23.175580 15832 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b8cbc143cac4cd8bc3e990221c4f28e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b8cbc143cac4cd8bc3e990221c4f28e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37025 } }
I20260924 04:11:23.176087 15827 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b8cbc143cac4cd8bc3e990221c4f28e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:11:23.177222 15833 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b8cbc143cac4cd8bc3e990221c4f28e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b8cbc143cac4cd8bc3e990221c4f28e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b8cbc143cac4cd8bc3e990221c4f28e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37025 } } }
I20260924 04:11:23.177361 15833 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b8cbc143cac4cd8bc3e990221c4f28e [sys.catalog]: This master's current role is: LEADER
I20260924 04:11:23.177724 15841 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:11:23.177722 15834 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b8cbc143cac4cd8bc3e990221c4f28e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b8cbc143cac4cd8bc3e990221c4f28e. Latest consensus state: current_term: 1 leader_uuid: "4b8cbc143cac4cd8bc3e990221c4f28e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b8cbc143cac4cd8bc3e990221c4f28e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37025 } } }
I20260924 04:11:23.178375 15834 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b8cbc143cac4cd8bc3e990221c4f28e [sys.catalog]: This master's current role is: LEADER
I20260924 04:11:23.180999 15841 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:11:23.186246 15841 catalog_manager.cc:1347] Generated new cluster ID: 5cd8e9d1e20a4497b23d8cb482326859
I20260924 04:11:23.186327 15841 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:11:23.195245 15841 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:11:23.196542 15841 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:11:23.213032 15841 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4b8cbc143cac4cd8bc3e990221c4f28e: Generated new TSK 0
I20260924 04:11:23.213896 15841 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:11:23.244253 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37025
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:11:23.362324 15853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:11:23.362619 15853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:11:23.362705 15853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:11:23.366755 15853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:11:23.366840 15853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:11:23.366927 15853 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:11:23.372195 15853 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15853
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:11:23.373270 15853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:11:23.374281 15853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:11:23.383808 15861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:11:23.383819 15859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:11:23.384001 15858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:11:23.384184 15853 server_base.cc:1034] running on GCE node
I20260924 04:11:23.384493 15853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:11:23.385141 15853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:11:23.386308 15853 hybrid_clock.cc:648] HybridClock initialized: now 1790223083386295 us; error 27 us; skew 500 ppm
Sep 24 04:11:23 dist-test-slave-00ld krb5kdc[15746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223083, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:11:23.388989 15853 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:11:23.389820 15853 webserver.cc:460] Webserver started at http://127.0.98.1:41211/ using document root <none> and password file <none>
I20260924 04:11:23.390354 15853 fs_manager.cc:359] Metadata directory not provided
I20260924 04:11:23.390453 15853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:11:23.390693 15853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:11:23.392627 15853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-0/data/instance:
uuid: "29fe56f4efe74850870cd033fdea81d3"
format_stamp: "Formatted at 2026-09-24 04:11:23 on dist-test-slave-00ld"
server_key: "a8a753b5000d70b1e941547fbea391b0"
server_key_iv: "ff1443f5b70c7947ef0f1f9a3e1a936b"
server_key_version: "encryptionkey@0"
I20260924 04:11:23.393256 15853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-0/wal/instance:
uuid: "29fe56f4efe74850870cd033fdea81d3"
format_stamp: "Formatted at 2026-09-24 04:11:23 on dist-test-slave-00ld"
server_key: "a8a753b5000d70b1e941547fbea391b0"
server_key_iv: "ff1443f5b70c7947ef0f1f9a3e1a936b"
server_key_version: "encryptionkey@0"
I20260924 04:11:23.396944 15853 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:11:23.399243 15868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:11:23.400451 15853 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:11:23.400542 15853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-0/wal
uuid: "29fe56f4efe74850870cd033fdea81d3"
format_stamp: "Formatted at 2026-09-24 04:11:23 on dist-test-slave-00ld"
server_key: "a8a753b5000d70b1e941547fbea391b0"
server_key_iv: "ff1443f5b70c7947ef0f1f9a3e1a936b"
server_key_version: "encryptionkey@0"
I20260924 04:11:23.400624 15853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:11:23.415863 15853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:11:23.419242 15853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:11:23.419457 15853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:11:23.420068 15853 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:11:23.420964 15853 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:11:23.421012 15853 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:11:23.421079 15853 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:11:23.421109 15853 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:11:23.430125 15853 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42515
I20260924 04:11:23.430145 15981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42515 every 8 connection(s)
I20260924 04:11:23.431005 15853 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:11:23.431577 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 15853
I20260924 04:11:23.431718 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:11:23.432085 392 external_mini_cluster.cc:1428] Setting key 828d799f2a275a9bc36b7e559489bb9a
Sep 24 04:11:23 dist-test-slave-00ld krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223083, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 04:11:23.445705 15830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:23.433066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53733 (local address 127.0.98.62:37025)
0924 04:11:23.433663 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0924 04:11:23.433667 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:11:23.434270 (+ 603us) server_negotiation.cc:407] Connection header received
0924 04:11:23.435561 (+ 1291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:11:23.435565 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:11:23.435617 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:11:23.435718 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:11:23.436908 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:23.437432 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:23.438582 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:23.438810 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:23.438937 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:11:23.441789 (+ 2852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:11:23.441793 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:11:23.441795 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:11:23.441819 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:11:23.443136 (+ 1317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:11:23.443649 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:11:23.443655 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:11:23.443659 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:11:23.443748 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:11:23.444072 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:11:23.444075 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:11:23.444076 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:11:23.444351 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:11:23.444491 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 04:11:23.445434 (+ 943us) server_negotiation.cc:299] Negotiation successful
0924 04:11:23.445570 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523}
I20260924 04:11:23.446830 15984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:23.433619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.1:53733)
0924 04:11:23.434126 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0924 04:11:23.434174 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 04:11:23.435350 (+ 1176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:11:23.435889 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:11:23.435896 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:11:23.436239 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:11:23.436769 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:11:23.436784 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:11:23.437569 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:11:23.437572 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:11:23.438451 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:11:23.438460 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:11:23.438985 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:11:23.438989 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:11:23.439147 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:11:23.439768 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:11:23.439783 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:11:23.441586 (+ 1803us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:11:23.443244 (+ 1658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:11:23.443249 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:11:23.443275 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:11:23.443532 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:11:23.443827 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:11:23.443829 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:11:23.443830 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:11:23.443962 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:11:23.444573 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:11:23.444580 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:11:23.445057 (+ 477us) client_negotiation.cc:769] Sending connection context
0924 04:11:23.445315 (+ 258us) client_negotiation.cc:240] Negotiation successful
0924 04:11:23.445727 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":180,"threads_started":1}
I20260924 04:11:23.448204 15982 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37025
I20260924 04:11:23.448487 15982 heartbeater.cc:464] Registering TS with master...
I20260924 04:11:23.449158 15982 heartbeater.cc:511] Master 127.0.98.62:37025 requested a full tablet report, sending...
I20260924 04:11:23.451097 15791 ts_manager.cc:194] Registered new tserver with Master: 29fe56f4efe74850870cd033fdea81d3 (127.0.98.1:42515)
I20260924 04:11:23.452893 15791 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:53733
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:11:23.464395 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37025
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:11:23.574730 15987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:11:23.574994 15987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:11:23.575037 15987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:11:23.579196 15987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:11:23.579252 15987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:11:23.579309 15987 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:11:23.584381 15987 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15987
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:11:23.585384 15987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:11:23.586329 15987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:11:23.595806 15993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:11:23.595826 15987 server_base.cc:1034] running on GCE node
W20260924 04:11:23.595791 15992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:11:23.595786 15995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:11:23.596529 15987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:11:23.597182 15987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:11:23.598374 15987 hybrid_clock.cc:648] HybridClock initialized: now 1790223083598338 us; error 49 us; skew 500 ppm
Sep 24 04:11:23 dist-test-slave-00ld krb5kdc[15746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223083, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:11:23.601116 15987 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:11:23.601990 15987 webserver.cc:460] Webserver started at http://127.0.98.2:36121/ using document root <none> and password file <none>
I20260924 04:11:23.602548 15987 fs_manager.cc:359] Metadata directory not provided
I20260924 04:11:23.602636 15987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:11:23.602872 15987 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:11:23.605118 15987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-1/data/instance:
uuid: "b30b39da74e84f2a8af94206e4a2e8e2"
format_stamp: "Formatted at 2026-09-24 04:11:23 on dist-test-slave-00ld"
server_key: "26eeade390c1ca46f79b67a27c04fa87"
server_key_iv: "edf98eef0f41d7c98248f520dee3a3b8"
server_key_version: "encryptionkey@0"
I20260924 04:11:23.605667 15987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b30b39da74e84f2a8af94206e4a2e8e2"
format_stamp: "Formatted at 2026-09-24 04:11:23 on dist-test-slave-00ld"
server_key: "26eeade390c1ca46f79b67a27c04fa87"
server_key_iv: "edf98eef0f41d7c98248f520dee3a3b8"
server_key_version: "encryptionkey@0"
I20260924 04:11:23.609875 15987 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260924 04:11:23.612310 16002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:11:23.613588 15987 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:11:23.613744 15987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-1/wal
uuid: "b30b39da74e84f2a8af94206e4a2e8e2"
format_stamp: "Formatted at 2026-09-24 04:11:23 on dist-test-slave-00ld"
server_key: "26eeade390c1ca46f79b67a27c04fa87"
server_key_iv: "edf98eef0f41d7c98248f520dee3a3b8"
server_key_version: "encryptionkey@0"
I20260924 04:11:23.613885 15987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:11:23.626529 15987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:11:23.629400 15987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:11:23.629549 15987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:11:23.630062 15987 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:11:23.630852 15987 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:11:23.630897 15987 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:11:23.630937 15987 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:11:23.630952 15987 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:11:23.639765 15987 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38987
I20260924 04:11:23.639786 16115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38987 every 8 connection(s)
I20260924 04:11:23.640765 15987 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:11:23.650872 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 15987
I20260924 04:11:23.651002 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:11:23.651283 392 external_mini_cluster.cc:1428] Setting key 0cc487c9baebe06cddb14d88562ed0ad
Sep 24 04:11:23 dist-test-slave-00ld krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223083, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:11:23.658944 15830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:23.643497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46549 (local address 127.0.98.62:37025)
0924 04:11:23.643694 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 04:11:23.643700 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:11:23.644990 (+ 1290us) server_negotiation.cc:407] Connection header received
0924 04:11:23.646903 (+ 1913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:11:23.646907 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:11:23.646964 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:11:23.647053 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:11:23.648541 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:23.649163 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:23.651660 (+ 2497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:23.651898 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:23.652020 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:11:23.654565 (+ 2545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:11:23.654571 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:11:23.654575 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:11:23.654608 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:11:23.656222 (+ 1614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:11:23.656920 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:11:23.656923 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:11:23.656924 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:11:23.656997 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:11:23.657437 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:11:23.657440 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:11:23.657441 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:11:23.657624 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:11:23.657700 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 04:11:23.658509 (+ 809us) server_negotiation.cc:299] Negotiation successful
0924 04:11:23.658748 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 04:11:23.659869 16118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:23.644056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.2:46549)
0924 04:11:23.644807 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0924 04:11:23.644875 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0924 04:11:23.646637 (+ 1762us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:11:23.647273 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:11:23.647285 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:11:23.647753 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:11:23.648390 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:11:23.648407 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:11:23.649300 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:11:23.649305 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:11:23.650531 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:11:23.650541 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:11:23.652059 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:11:23.652063 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:11:23.652250 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:11:23.652898 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:11:23.652915 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:11:23.654391 (+ 1476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:11:23.656347 (+ 1956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:11:23.656356 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:11:23.656377 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:11:23.656818 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:11:23.657095 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:11:23.657098 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:11:23.657102 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:11:23.657326 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:11:23.657739 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:11:23.657750 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:11:23.658116 (+ 366us) client_negotiation.cc:769] Sending connection context
0924 04:11:23.658368 (+ 252us) client_negotiation.cc:240] Negotiation successful
0924 04:11:23.658762 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":172,"threads_started":1}
I20260924 04:11:23.661201 16116 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37025
I20260924 04:11:23.661455 16116 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 04:11:23.662003 16116 heartbeater.cc:511] Master 127.0.98.62:37025 requested a full tablet report, sending...
I20260924 04:11:23.663084 15791 ts_manager.cc:194] Registered new tserver with Master: b30b39da74e84f2a8af94206e4a2e8e2 (127.0.98.2:38987)
I20260924 04:11:23.663672 15791 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:46549
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:11:23.680020 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37025
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:11:23.790807 16121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:11:23.791074 16121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:11:23.791117 16121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:11:23.795110 16121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:11:23.795168 16121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:11:23.795269 16121 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:11:23.800485 16121 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16121
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:11:23.801584 16121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:11:23.802651 16121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:11:23.812361 16127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:11:23.812392 16126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:11:23.812546 16121 server_base.cc:1034] running on GCE node
W20260924 04:11:23.812361 16129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:11:23.813122 16121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:11:23.813868 16121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:11:23.815073 16121 hybrid_clock.cc:648] HybridClock initialized: now 1790223083815055 us; error 33 us; skew 500 ppm
Sep 24 04:11:23 dist-test-slave-00ld krb5kdc[15746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223083, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:11:23.818077 16121 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:11:23.819089 16121 webserver.cc:460] Webserver started at http://127.0.98.3:39903/ using document root <none> and password file <none>
I20260924 04:11:23.819670 16121 fs_manager.cc:359] Metadata directory not provided
I20260924 04:11:23.819730 16121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:11:23.819941 16121 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:11:23.822016 16121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-2/data/instance:
uuid: "59864430ab9443eb82dc451f4a293582"
format_stamp: "Formatted at 2026-09-24 04:11:23 on dist-test-slave-00ld"
server_key: "1203e1471249287047c88de5eb16b603"
server_key_iv: "67fc4e595c5546b149140b244bc26258"
server_key_version: "encryptionkey@0"
I20260924 04:11:23.822570 16121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-2/wal/instance:
uuid: "59864430ab9443eb82dc451f4a293582"
format_stamp: "Formatted at 2026-09-24 04:11:23 on dist-test-slave-00ld"
server_key: "1203e1471249287047c88de5eb16b603"
server_key_iv: "67fc4e595c5546b149140b244bc26258"
server_key_version: "encryptionkey@0"
I20260924 04:11:23.826256 16121 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260924 04:11:23.828661 16136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:11:23.829977 16121 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 04:11:23.830111 16121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-2/wal
uuid: "59864430ab9443eb82dc451f4a293582"
format_stamp: "Formatted at 2026-09-24 04:11:23 on dist-test-slave-00ld"
server_key: "1203e1471249287047c88de5eb16b603"
server_key_iv: "67fc4e595c5546b149140b244bc26258"
server_key_version: "encryptionkey@0"
I20260924 04:11:23.830240 16121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:11:23.844514 16121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:11:23.847527 16121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:11:23.847733 16121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:11:23.848331 16121 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:11:23.849153 16121 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:11:23.849221 16121 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:11:23.849292 16121 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:11:23.849342 16121 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:11:23.858021 16121 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42569
I20260924 04:11:23.858058 16249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42569 every 8 connection(s)
I20260924 04:11:23.858919 16121 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:11:23.866147 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 16121
I20260924 04:11:23.866295 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:11:23.866602 392 external_mini_cluster.cc:1428] Setting key 3829cb6d3863025a6de2a7cfc13c9c29
Sep 24 04:11:23 dist-test-slave-00ld krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223083, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:11:23.872107 15830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:23.860711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45657 (local address 127.0.98.62:37025)
0924 04:11:23.860858 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 04:11:23.860863 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:11:23.861735 (+ 872us) server_negotiation.cc:407] Connection header received
0924 04:11:23.862969 (+ 1234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:11:23.862972 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:11:23.863016 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:11:23.863089 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:11:23.864197 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:23.864801 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:23.865875 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:23.866101 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:23.866231 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:11:23.869147 (+ 2916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:11:23.869150 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:11:23.869152 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:11:23.869175 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:11:23.870380 (+ 1205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:11:23.870814 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:11:23.870816 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:11:23.870818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:11:23.870908 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:11:23.871189 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:11:23.871191 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:11:23.871192 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:11:23.871363 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:11:23.871437 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 04:11:23.871865 (+ 428us) server_negotiation.cc:299] Negotiation successful
0924 04:11:23.871988 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 04:11:23.872954 16252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:23.861154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.3:45657)
0924 04:11:23.861584 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0924 04:11:23.861624 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 04:11:23.862775 (+ 1151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:11:23.863234 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:11:23.863241 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:11:23.863604 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:11:23.864067 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:11:23.864080 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:11:23.864900 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:11:23.864902 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:11:23.865764 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:11:23.865773 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:11:23.866262 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:11:23.866266 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:11:23.866483 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:11:23.867138 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:11:23.867163 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:11:23.868975 (+ 1812us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:11:23.870477 (+ 1502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:11:23.870481 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:11:23.870492 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:11:23.870731 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:11:23.870985 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:11:23.870987 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:11:23.870989 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:11:23.871112 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:11:23.871448 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:11:23.871452 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:11:23.871661 (+ 209us) client_negotiation.cc:769] Sending connection context
0924 04:11:23.871805 (+ 144us) client_negotiation.cc:240] Negotiation successful
0924 04:11:23.872037 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":114,"threads_started":1}
I20260924 04:11:23.873978 16250 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37025
I20260924 04:11:23.874217 16250 heartbeater.cc:464] Registering TS with master...
I20260924 04:11:23.874714 16250 heartbeater.cc:511] Master 127.0.98.62:37025 requested a full tablet report, sending...
I20260924 04:11:23.875689 15791 ts_manager.cc:194] Registered new tserver with Master: 59864430ab9443eb82dc451f4a293582 (127.0.98.3:42569)
I20260924 04:11:23.876262 15791 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45657
I20260924 04:11:23.880373 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:11:23.889756 15830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:23.882353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59998 (local address 127.0.98.62:37025)
0924 04:11:23.882487 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 04:11:23.882492 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:11:23.882674 (+ 182us) server_negotiation.cc:407] Connection header received
0924 04:11:23.882766 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:11:23.882768 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:11:23.882805 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:11:23.882866 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:11:23.883521 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:23.884206 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:23.885249 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:23.885492 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:23.885601 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:11:23.886865 (+ 1264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:11:23.886869 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:11:23.886871 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:11:23.886896 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:11:23.888126 (+ 1230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:11:23.888564 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:11:23.888569 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:11:23.888571 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:11:23.888654 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:11:23.888955 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:11:23.888957 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:11:23.888959 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:11:23.889156 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:11:23.889257 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 04:11:23.889465 (+ 208us) server_negotiation.cc:299] Negotiation successful
0924 04:11:23.889560 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 04:11:23.892856 15791 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59998:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:11:23.895853 15791 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:11:23.910153 16262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:23.904490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42515 (local address 127.0.0.1:51420)
0924 04:11:23.905206 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0924 04:11:23.905220 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:11:23.905297 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:11:23.906333 (+ 1036us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:11:23.906335 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:11:23.906350 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:11:23.906429 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:11:23.906433 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:11:23.907434 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:11:23.907439 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:11:23.908621 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:11:23.908630 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:11:23.909587 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:11:23.909591 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:11:23.909696 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:11:23.909735 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 04:11:23.909855 (+ 120us) client_negotiation.cc:240] Negotiation successful
0924 04:11:23.909974 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":656,"thread_start_us":88,"threads_started":1}
I20260924 04:11:23.910162 16266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:23.905391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51420 (local address 127.0.98.1:42515)
0924 04:11:23.905669 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0924 04:11:23.905682 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 04:11:23.905704 (+ 22us) server_negotiation.cc:407] Connection header received
0924 04:11:23.905778 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:11:23.905784 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:11:23.905974 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:11:23.906164 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:11:23.906541 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:23.907298 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:23.908781 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:23.909458 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:23.909564 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:11:23.909716 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 04:11:23.909898 (+ 182us) server_negotiation.cc:299] Negotiation successful
0924 04:11:23.909992 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":124,"threads_started":1}
I20260924 04:11:23.910530 16261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:23.904031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42569 (local address 127.0.0.1:37318)
0924 04:11:23.904783 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0924 04:11:23.904846 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0924 04:11:23.904967 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:11:23.905930 (+ 963us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:11:23.905936 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:11:23.905957 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:11:23.906132 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:11:23.906142 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:11:23.907738 (+ 1596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:11:23.907741 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:11:23.909120 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:11:23.909131 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:11:23.910131 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:11:23.910134 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:11:23.910233 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:11:23.910266 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 04:11:23.910317 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 04:11:23.910392 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"mutex_wait_us":112,"thread_start_us":80,"threads_started":1}
I20260924 04:11:23.911049 16263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:23.904203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37318 (local address 127.0.98.3:42569)
0924 04:11:23.905525 (+ 1322us) server_negotiation.cc:206] Beginning negotiation
0924 04:11:23.905534 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 04:11:23.905552 (+ 18us) server_negotiation.cc:407] Connection header received
0924 04:11:23.905614 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:11:23.905619 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:11:23.905773 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:11:23.905943 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:11:23.906271 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:23.907612 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:23.909290 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:23.910006 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:23.910730 (+ 724us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:11:23.910837 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 04:11:23.910877 (+ 40us) server_negotiation.cc:299] Negotiation successful
0924 04:11:23.910943 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1236,"thread_start_us":146,"threads_started":1}
I20260924 04:11:23.911691 16264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:23.904244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38987 (local address 127.0.0.1:53882)
0924 04:11:23.905095 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0924 04:11:23.905109 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:11:23.905199 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:11:23.906163 (+ 964us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:11:23.906168 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:11:23.906190 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:11:23.906294 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:11:23.906300 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:11:23.907641 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:11:23.907644 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:11:23.909121 (+ 1477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:11:23.909131 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:11:23.910187 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:11:23.910191 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:11:23.911427 (+ 1236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:11:23.911452 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 04:11:23.911506 (+ 54us) client_negotiation.cc:240] Negotiation successful
0924 04:11:23.911571 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":793,"mutex_wait_us":107,"thread_start_us":104,"threads_started":1}
I20260924 04:11:23.912364 16265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:23.904913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53882 (local address 127.0.98.2:38987)
0924 04:11:23.905788 (+ 875us) server_negotiation.cc:206] Beginning negotiation
0924 04:11:23.905798 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 04:11:23.905816 (+ 18us) server_negotiation.cc:407] Connection header received
0924 04:11:23.905876 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:11:23.905891 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:11:23.906035 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:11:23.906166 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:11:23.906422 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:23.907494 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:23.909249 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:23.910048 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:23.911968 (+ 1920us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:11:23.912098 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0924 04:11:23.912144 (+ 46us) server_negotiation.cc:299] Negotiation successful
0924 04:11:23.912228 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":788,"thread_start_us":100,"threads_started":1}
I20260924 04:11:23.913369 15916 tablet_service.cc:1467] Processing CreateTablet for tablet 1ef81ae30bd7496787327eec67f0a089 (DEFAULT_TABLE table=test-table [id=d1007511c3904fbab77fd65e4572c0a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:11:23.913748 16184 tablet_service.cc:1467] Processing CreateTablet for tablet 1ef81ae30bd7496787327eec67f0a089 (DEFAULT_TABLE table=test-table [id=d1007511c3904fbab77fd65e4572c0a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:11:23.914419 15916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ef81ae30bd7496787327eec67f0a089. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:11:23.914769 16184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ef81ae30bd7496787327eec67f0a089. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:11:23.914927 16050 tablet_service.cc:1467] Processing CreateTablet for tablet 1ef81ae30bd7496787327eec67f0a089 (DEFAULT_TABLE table=test-table [id=d1007511c3904fbab77fd65e4572c0a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:11:23.915912 16050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ef81ae30bd7496787327eec67f0a089. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:11:23.921312 16267 tablet_bootstrap.cc:492] T 1ef81ae30bd7496787327eec67f0a089 P 29fe56f4efe74850870cd033fdea81d3: Bootstrap starting.
I20260924 04:11:23.922497 16268 tablet_bootstrap.cc:492] T 1ef81ae30bd7496787327eec67f0a089 P 59864430ab9443eb82dc451f4a293582: Bootstrap starting.
I20260924 04:11:23.922539 16269 tablet_bootstrap.cc:492] T 1ef81ae30bd7496787327eec67f0a089 P b30b39da74e84f2a8af94206e4a2e8e2: Bootstrap starting.
I20260924 04:11:23.923621 16267 tablet_bootstrap.cc:654] T 1ef81ae30bd7496787327eec67f0a089 P 29fe56f4efe74850870cd033fdea81d3: Neither blocks nor log segments found. Creating new log.
I20260924 04:11:23.924306 16268 tablet_bootstrap.cc:654] T 1ef81ae30bd7496787327eec67f0a089 P 59864430ab9443eb82dc451f4a293582: Neither blocks nor log segments found. Creating new log.
I20260924 04:11:23.925016 16269 tablet_bootstrap.cc:654] T 1ef81ae30bd7496787327eec67f0a089 P b30b39da74e84f2a8af94206e4a2e8e2: Neither blocks nor log segments found. Creating new log.
I20260924 04:11:23.924985 16267 log.cc:824] T 1ef81ae30bd7496787327eec67f0a089 P 29fe56f4efe74850870cd033fdea81d3: Log is configured to *not* fsync() on all Append() calls
I20260924 04:11:23.925410 16268 log.cc:824] T 1ef81ae30bd7496787327eec67f0a089 P 59864430ab9443eb82dc451f4a293582: Log is configured to *not* fsync() on all Append() calls
I20260924 04:11:23.925804 16269 log.cc:824] T 1ef81ae30bd7496787327eec67f0a089 P b30b39da74e84f2a8af94206e4a2e8e2: Log is configured to *not* fsync() on all Append() calls
I20260924 04:11:23.927094 16267 tablet_bootstrap.cc:492] T 1ef81ae30bd7496787327eec67f0a089 P 29fe56f4efe74850870cd033fdea81d3: No bootstrap required, opened a new log
I20260924 04:11:23.927191 16268 tablet_bootstrap.cc:492] T 1ef81ae30bd7496787327eec67f0a089 P 59864430ab9443eb82dc451f4a293582: No bootstrap required, opened a new log
I20260924 04:11:23.927307 16267 ts_tablet_manager.cc:1397] T 1ef81ae30bd7496787327eec67f0a089 P 29fe56f4efe74850870cd033fdea81d3: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20260924 04:11:23.927402 16268 ts_tablet_manager.cc:1397] T 1ef81ae30bd7496787327eec67f0a089 P 59864430ab9443eb82dc451f4a293582: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260924 04:11:23.930173 16267 raft_consensus.cc:348] T 1ef81ae30bd7496787327eec67f0a089 P 29fe56f4efe74850870cd033fdea81d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59864430ab9443eb82dc451f4a293582" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42569 } } peers { permanent_uuid: "b30b39da74e84f2a8af94206e4a2e8e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38987 } } peers { permanent_uuid: "29fe56f4efe74850870cd033fdea81d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42515 } }
I20260924 04:11:23.930171 16268 raft_consensus.cc:348] T 1ef81ae30bd7496787327eec67f0a089 P 59864430ab9443eb82dc451f4a293582 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59864430ab9443eb82dc451f4a293582" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42569 } } peers { permanent_uuid: "b30b39da74e84f2a8af94206e4a2e8e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38987 } } peers { permanent_uuid: "29fe56f4efe74850870cd033fdea81d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42515 } }
I20260924 04:11:23.930408 16267 raft_consensus.cc:374] T 1ef81ae30bd7496787327eec67f0a089 P 29fe56f4efe74850870cd033fdea81d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:11:23.930431 16268 raft_consensus.cc:374] T 1ef81ae30bd7496787327eec67f0a089 P 59864430ab9443eb82dc451f4a293582 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:11:23.930475 16267 raft_consensus.cc:729] T 1ef81ae30bd7496787327eec67f0a089 P 29fe56f4efe74850870cd033fdea81d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29fe56f4efe74850870cd033fdea81d3, State: Initialized, Role: FOLLOWER
I20260924 04:11:23.930493 16268 raft_consensus.cc:729] T 1ef81ae30bd7496787327eec67f0a089 P 59864430ab9443eb82dc451f4a293582 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59864430ab9443eb82dc451f4a293582, State: Initialized, Role: FOLLOWER
I20260924 04:11:23.930948 16268 consensus_queue.cc:260] T 1ef81ae30bd7496787327eec67f0a089 P 59864430ab9443eb82dc451f4a293582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59864430ab9443eb82dc451f4a293582" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42569 } } peers { permanent_uuid: "b30b39da74e84f2a8af94206e4a2e8e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38987 } } peers { permanent_uuid: "29fe56f4efe74850870cd033fdea81d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42515 } }
I20260924 04:11:23.930954 16267 consensus_queue.cc:260] T 1ef81ae30bd7496787327eec67f0a089 P 29fe56f4efe74850870cd033fdea81d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59864430ab9443eb82dc451f4a293582" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42569 } } peers { permanent_uuid: "b30b39da74e84f2a8af94206e4a2e8e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38987 } } peers { permanent_uuid: "29fe56f4efe74850870cd033fdea81d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42515 } }
I20260924 04:11:23.931676 16250 heartbeater.cc:503] Master 127.0.98.62:37025 was elected leader, sending a full tablet report...
I20260924 04:11:23.931939 16267 ts_tablet_manager.cc:1428] T 1ef81ae30bd7496787327eec67f0a089 P 29fe56f4efe74850870cd033fdea81d3: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
I20260924 04:11:23.931939 16268 ts_tablet_manager.cc:1428] T 1ef81ae30bd7496787327eec67f0a089 P 59864430ab9443eb82dc451f4a293582: Time spent starting tablet: real 0.004s user 0.004s sys 0.001s
W20260924 04:11:23.932153 15983 tablet.cc:2369] T 1ef81ae30bd7496787327eec67f0a089 P 29fe56f4efe74850870cd033fdea81d3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:11:23.932366 16269 tablet_bootstrap.cc:492] T 1ef81ae30bd7496787327eec67f0a089 P b30b39da74e84f2a8af94206e4a2e8e2: No bootstrap required, opened a new log
I20260924 04:11:23.932556 16269 ts_tablet_manager.cc:1397] T 1ef81ae30bd7496787327eec67f0a089 P b30b39da74e84f2a8af94206e4a2e8e2: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260924 04:11:23.933144 15982 heartbeater.cc:503] Master 127.0.98.62:37025 was elected leader, sending a full tablet report...
I20260924 04:11:23.937094 16269 raft_consensus.cc:348] T 1ef81ae30bd7496787327eec67f0a089 P b30b39da74e84f2a8af94206e4a2e8e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59864430ab9443eb82dc451f4a293582" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42569 } } peers { permanent_uuid: "b30b39da74e84f2a8af94206e4a2e8e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38987 } } peers { permanent_uuid: "29fe56f4efe74850870cd033fdea81d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42515 } }
I20260924 04:11:23.937354 16269 raft_consensus.cc:374] T 1ef81ae30bd7496787327eec67f0a089 P b30b39da74e84f2a8af94206e4a2e8e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:11:23.937440 16269 raft_consensus.cc:729] T 1ef81ae30bd7496787327eec67f0a089 P b30b39da74e84f2a8af94206e4a2e8e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b30b39da74e84f2a8af94206e4a2e8e2, State: Initialized, Role: FOLLOWER
I20260924 04:11:23.937824 16269 consensus_queue.cc:260] T 1ef81ae30bd7496787327eec67f0a089 P b30b39da74e84f2a8af94206e4a2e8e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59864430ab9443eb82dc451f4a293582" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42569 } } peers { permanent_uuid: "b30b39da74e84f2a8af94206e4a2e8e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38987 } } peers { permanent_uuid: "29fe56f4efe74850870cd033fdea81d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42515 } }
I20260924 04:11:23.938426 16116 heartbeater.cc:503] Master 127.0.98.62:37025 was elected leader, sending a full tablet report...
I20260924 04:11:23.938668 16269 ts_tablet_manager.cc:1428] T 1ef81ae30bd7496787327eec67f0a089 P b30b39da74e84f2a8af94206e4a2e8e2: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260924 04:11:24.033304 16275 raft_consensus.cc:482] T 1ef81ae30bd7496787327eec67f0a089 P b30b39da74e84f2a8af94206e4a2e8e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:11:24.033602 16275 raft_consensus.cc:504] T 1ef81ae30bd7496787327eec67f0a089 P b30b39da74e84f2a8af94206e4a2e8e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59864430ab9443eb82dc451f4a293582" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42569 } } peers { permanent_uuid: "b30b39da74e84f2a8af94206e4a2e8e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38987 } } peers { permanent_uuid: "29fe56f4efe74850870cd033fdea81d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42515 } }
I20260924 04:11:24.034935 16275 leader_election.cc:288] T 1ef81ae30bd7496787327eec67f0a089 P b30b39da74e84f2a8af94206e4a2e8e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59864430ab9443eb82dc451f4a293582 (127.0.98.3:42569), 29fe56f4efe74850870cd033fdea81d3 (127.0.98.1:42515)
I20260924 04:11:24.038553 16118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:24.035208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42569 (local address 127.0.98.2:44755)
0924 04:11:24.035346 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 04:11:24.035365 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 04:11:24.035474 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:11:24.035760 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:11:24.035763 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:11:24.035779 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:11:24.035902 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:11:24.035914 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:11:24.036650 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:11:24.036654 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:11:24.037457 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:11:24.037466 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:11:24.038115 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:11:24.038119 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:11:24.038225 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:11:24.038253 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 04:11:24.038348 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 04:11:24.038422 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260924 04:11:24.038554 16263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:24.035310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44755 (local address 127.0.98.3:42569)
0924 04:11:24.035450 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 04:11:24.035454 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:11:24.035469 (+ 15us) server_negotiation.cc:407] Connection header received
0924 04:11:24.035597 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:11:24.035605 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:11:24.035650 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:11:24.035749 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:11:24.036026 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:24.036535 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:24.037588 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:24.037996 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:24.038112 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:11:24.038174 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 04:11:24.038365 (+ 191us) server_negotiation.cc:299] Negotiation successful
0924 04:11:24.038431 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 04:11:24.038820 16266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:24.035332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52271 (local address 127.0.98.1:42515)
0924 04:11:24.035469 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 04:11:24.035475 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:11:24.035816 (+ 341us) server_negotiation.cc:407] Connection header received
0924 04:11:24.035976 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:11:24.035979 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:11:24.036030 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:11:24.036114 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:11:24.036358 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:24.036928 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:24.037915 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:24.038292 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:24.038407 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:11:24.038466 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 04:11:24.038641 (+ 175us) server_negotiation.cc:299] Negotiation successful
0924 04:11:24.038713 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 04:11:24.038833 16276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:24.035208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42515 (local address 127.0.98.2:52271)
0924 04:11:24.035724 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0924 04:11:24.035735 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:11:24.035854 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:11:24.036124 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:11:24.036127 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:11:24.036142 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:11:24.036254 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:11:24.036259 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:11:24.037035 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:11:24.037038 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:11:24.037792 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:11:24.037800 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:11:24.038415 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:11:24.038418 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:11:24.038519 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:11:24.038542 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 04:11:24.038624 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 04:11:24.038688 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"mutex_wait_us":172,"thread_start_us":165,"threads_started":1}
I20260924 04:11:24.039250 16204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ef81ae30bd7496787327eec67f0a089" candidate_uuid: "b30b39da74e84f2a8af94206e4a2e8e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59864430ab9443eb82dc451f4a293582" is_pre_election: true
I20260924 04:11:24.039565 15931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ef81ae30bd7496787327eec67f0a089" candidate_uuid: "b30b39da74e84f2a8af94206e4a2e8e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29fe56f4efe74850870cd033fdea81d3" is_pre_election: true
I20260924 04:11:24.039618 16204 raft_consensus.cc:2454] T 1ef81ae30bd7496787327eec67f0a089 P 59864430ab9443eb82dc451f4a293582 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b30b39da74e84f2a8af94206e4a2e8e2 in term 0.
I20260924 04:11:24.039851 15931 raft_consensus.cc:2454] T 1ef81ae30bd7496787327eec67f0a089 P 29fe56f4efe74850870cd033fdea81d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b30b39da74e84f2a8af94206e4a2e8e2 in term 0.
I20260924 04:11:24.040103 16003 leader_election.cc:302] T 1ef81ae30bd7496787327eec67f0a089 P b30b39da74e84f2a8af94206e4a2e8e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59864430ab9443eb82dc451f4a293582, b30b39da74e84f2a8af94206e4a2e8e2; no voters:
I20260924 04:11:24.040401 16275 raft_consensus.cc:2781] T 1ef81ae30bd7496787327eec67f0a089 P b30b39da74e84f2a8af94206e4a2e8e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:11:24.040498 16275 raft_consensus.cc:482] T 1ef81ae30bd7496787327eec67f0a089 P b30b39da74e84f2a8af94206e4a2e8e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:11:24.040577 16275 raft_consensus.cc:3037] T 1ef81ae30bd7496787327eec67f0a089 P b30b39da74e84f2a8af94206e4a2e8e2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:11:24.041652 16275 raft_consensus.cc:504] T 1ef81ae30bd7496787327eec67f0a089 P b30b39da74e84f2a8af94206e4a2e8e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59864430ab9443eb82dc451f4a293582" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42569 } } peers { permanent_uuid: "b30b39da74e84f2a8af94206e4a2e8e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38987 } } peers { permanent_uuid: "29fe56f4efe74850870cd033fdea81d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42515 } }
I20260924 04:11:24.041997 16275 leader_election.cc:288] T 1ef81ae30bd7496787327eec67f0a089 P b30b39da74e84f2a8af94206e4a2e8e2 [CANDIDATE]: Term 1 election: Requested vote from peers 59864430ab9443eb82dc451f4a293582 (127.0.98.3:42569), 29fe56f4efe74850870cd033fdea81d3 (127.0.98.1:42515)
I20260924 04:11:24.042373 15931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ef81ae30bd7496787327eec67f0a089" candidate_uuid: "b30b39da74e84f2a8af94206e4a2e8e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29fe56f4efe74850870cd033fdea81d3"
I20260924 04:11:24.042373 16204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ef81ae30bd7496787327eec67f0a089" candidate_uuid: "b30b39da74e84f2a8af94206e4a2e8e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59864430ab9443eb82dc451f4a293582"
I20260924 04:11:24.042500 16204 raft_consensus.cc:3037] T 1ef81ae30bd7496787327eec67f0a089 P 59864430ab9443eb82dc451f4a293582 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:11:24.042500 15931 raft_consensus.cc:3037] T 1ef81ae30bd7496787327eec67f0a089 P 29fe56f4efe74850870cd033fdea81d3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:11:24.043735 16204 raft_consensus.cc:2454] T 1ef81ae30bd7496787327eec67f0a089 P 59864430ab9443eb82dc451f4a293582 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b30b39da74e84f2a8af94206e4a2e8e2 in term 1.
I20260924 04:11:24.043735 15931 raft_consensus.cc:2454] T 1ef81ae30bd7496787327eec67f0a089 P 29fe56f4efe74850870cd033fdea81d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b30b39da74e84f2a8af94206e4a2e8e2 in term 1.
I20260924 04:11:24.044068 16006 leader_election.cc:302] T 1ef81ae30bd7496787327eec67f0a089 P b30b39da74e84f2a8af94206e4a2e8e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29fe56f4efe74850870cd033fdea81d3, b30b39da74e84f2a8af94206e4a2e8e2; no voters:
I20260924 04:11:24.044294 16275 raft_consensus.cc:2781] T 1ef81ae30bd7496787327eec67f0a089 P b30b39da74e84f2a8af94206e4a2e8e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:11:24.044530 16275 raft_consensus.cc:686] T 1ef81ae30bd7496787327eec67f0a089 P b30b39da74e84f2a8af94206e4a2e8e2 [term 1 LEADER]: Becoming Leader. State: Replica: b30b39da74e84f2a8af94206e4a2e8e2, State: Running, Role: LEADER
I20260924 04:11:24.044898 16275 consensus_queue.cc:237] T 1ef81ae30bd7496787327eec67f0a089 P b30b39da74e84f2a8af94206e4a2e8e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59864430ab9443eb82dc451f4a293582" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42569 } } peers { permanent_uuid: "b30b39da74e84f2a8af94206e4a2e8e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38987 } } peers { permanent_uuid: "29fe56f4efe74850870cd033fdea81d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42515 } }
I20260924 04:11:24.048143 15789 catalog_manager.cc:5495] T 1ef81ae30bd7496787327eec67f0a089 P b30b39da74e84f2a8af94206e4a2e8e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to b30b39da74e84f2a8af94206e4a2e8e2 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "b30b39da74e84f2a8af94206e4a2e8e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59864430ab9443eb82dc451f4a293582" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b30b39da74e84f2a8af94206e4a2e8e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38987 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "29fe56f4efe74850870cd033fdea81d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42515 } health_report { overall_health: UNKNOWN } } }
I20260924 04:11:24.062832 16265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:24.059521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53886 (local address 127.0.98.2:38987)
0924 04:11:24.059639 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0924 04:11:24.059642 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:11:24.059654 (+ 12us) server_negotiation.cc:407] Connection header received
0924 04:11:24.059775 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:11:24.059778 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:11:24.059828 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:11:24.059920 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:11:24.060116 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:24.060639 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:24.061549 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:24.061754 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:24.061831 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:11:24.062079 (+ 248us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:11:24.062499 (+ 420us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:11:24.062595 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 04:11:24.062651 (+ 56us) server_negotiation.cc:299] Negotiation successful
0924 04:11:24.062698 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 04:11:24.068881 16204 raft_consensus.cc:1261] T 1ef81ae30bd7496787327eec67f0a089 P 59864430ab9443eb82dc451f4a293582 [term 1 FOLLOWER]: Refusing update from remote peer b30b39da74e84f2a8af94206e4a2e8e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:11:24.068885 15931 raft_consensus.cc:1261] T 1ef81ae30bd7496787327eec67f0a089 P 29fe56f4efe74850870cd033fdea81d3 [term 1 FOLLOWER]: Refusing update from remote peer b30b39da74e84f2a8af94206e4a2e8e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:11:24.069585 16275 consensus_queue.cc:1035] T 1ef81ae30bd7496787327eec67f0a089 P b30b39da74e84f2a8af94206e4a2e8e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29fe56f4efe74850870cd033fdea81d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:11:24.069744 16277 consensus_queue.cc:1035] T 1ef81ae30bd7496787327eec67f0a089 P b30b39da74e84f2a8af94206e4a2e8e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59864430ab9443eb82dc451f4a293582" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:11:24.077087 16281 mvcc.cc:205] Tried to move back new op lower bound from 7332753752337924096 to 7332753752250273792. Current Snapshot: MvccSnapshot[applied={T|T < 7332753752337924096}]
I20260924 04:11:24.085237 15791 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59998:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:11:24.085409 15791 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59998:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:11:24.087612 15791 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4b8cbc143cac4cd8bc3e990221c4f28e: Sending DeleteTablet for 3 replicas of tablet 1ef81ae30bd7496787327eec67f0a089
I20260924 04:11:24.088364 16184 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ef81ae30bd7496787327eec67f0a089 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:11:24 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37318
I20260924 04:11:24.088406 15916 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ef81ae30bd7496787327eec67f0a089 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:11:24 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51420
I20260924 04:11:24.088723 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 15853
I20260924 04:11:24.088876 16050 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ef81ae30bd7496787327eec67f0a089 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:11:24 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53882
I20260924 04:11:24.090564 16291 tablet_replica.cc:330] T 1ef81ae30bd7496787327eec67f0a089 P b30b39da74e84f2a8af94206e4a2e8e2: stopping tablet replica
I20260924 04:11:24.090864 16291 raft_consensus.cc:2229] T 1ef81ae30bd7496787327eec67f0a089 P b30b39da74e84f2a8af94206e4a2e8e2 [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:11:24.091215 16291 raft_consensus.cc:2258] T 1ef81ae30bd7496787327eec67f0a089 P b30b39da74e84f2a8af94206e4a2e8e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:11:24.092450 16291 ts_tablet_manager.cc:1905] T 1ef81ae30bd7496787327eec67f0a089 P b30b39da74e84f2a8af94206e4a2e8e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:11:24.094660 16291 ts_tablet_manager.cc:1918] T 1ef81ae30bd7496787327eec67f0a089 P b30b39da74e84f2a8af94206e4a2e8e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:11:24.094771 16291 log.cc:1196] T 1ef81ae30bd7496787327eec67f0a089 P b30b39da74e84f2a8af94206e4a2e8e2: Deleting WAL directory at /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-1/wal/wals/1ef81ae30bd7496787327eec67f0a089
I20260924 04:11:24.095108 16291 ts_tablet_manager.cc:1939] T 1ef81ae30bd7496787327eec67f0a089 P b30b39da74e84f2a8af94206e4a2e8e2: Deleting consensus metadata
I20260924 04:11:24.096131 16289 tablet_replica.cc:330] T 1ef81ae30bd7496787327eec67f0a089 P 59864430ab9443eb82dc451f4a293582: stopping tablet replica
I20260924 04:11:24.096202 15777 catalog_manager.cc:4841] TS b30b39da74e84f2a8af94206e4a2e8e2 (127.0.98.2:38987): tablet 1ef81ae30bd7496787327eec67f0a089 (table test-table [id=d1007511c3904fbab77fd65e4572c0a1]) successfully deleted
W20260924 04:11:24.096267 15778 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42515 (error 108)
I20260924 04:11:24.096477 16289 raft_consensus.cc:2229] T 1ef81ae30bd7496787327eec67f0a089 P 59864430ab9443eb82dc451f4a293582 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:11:24.096822 16289 raft_consensus.cc:2258] T 1ef81ae30bd7496787327eec67f0a089 P 59864430ab9443eb82dc451f4a293582 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:11:24.097182 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 15987
I20260924 04:11:24.097517 16264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:24.097126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42515 (local address 127.0.0.1:51434)
0924 04:11:24.097314 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0924 04:11:24.097394 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
W20260924 04:11:24.098205 15778 catalog_manager.cc:4584] TS 29fe56f4efe74850870cd033fdea81d3 (127.0.98.1:42515): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1ef81ae30bd7496787327eec67f0a089: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42515: connect: Connection refused (error 111)
I20260924 04:11:24.099462 16289 ts_tablet_manager.cc:1905] T 1ef81ae30bd7496787327eec67f0a089 P 59864430ab9443eb82dc451f4a293582: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:11:24.101555 16289 ts_tablet_manager.cc:1918] T 1ef81ae30bd7496787327eec67f0a089 P 59864430ab9443eb82dc451f4a293582: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:11:24.101667 16289 log.cc:1196] T 1ef81ae30bd7496787327eec67f0a089 P 59864430ab9443eb82dc451f4a293582: Deleting WAL directory at /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-2/wal/wals/1ef81ae30bd7496787327eec67f0a089
I20260924 04:11:24.101971 16289 ts_tablet_manager.cc:1939] T 1ef81ae30bd7496787327eec67f0a089 P 59864430ab9443eb82dc451f4a293582: Deleting consensus metadata
I20260924 04:11:24.103137 15775 catalog_manager.cc:4841] TS 59864430ab9443eb82dc451f4a293582 (127.0.98.3:42569): tablet 1ef81ae30bd7496787327eec67f0a089 (table test-table [id=d1007511c3904fbab77fd65e4572c0a1]) successfully deleted
I20260924 04:11:24.104033 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 16121
I20260924 04:11:24.109743 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 15759
I20260924 04:11:44.117291 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37025
--webserver_interface=127.0.98.62
--webserver_port=37799
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37025
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:11:44.231676 16293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:11:44.231945 16293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:11:44.232012 16293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:11:44.236117 16293 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 04:11:44.236354 16293 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 04:11:44.236374 16293 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 04:11:44.236389 16293 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:11:44.236464 16293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:11:44.236490 16293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:11:44.236523 16293 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:11:44.236558 16293 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:11:44.242296 16293 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37025
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=37799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16293
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:11:44.243506 16293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:11:44.244593 16293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:11:44.257786 16300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:11:44.257813 16299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:11:44.257804 16302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:11:44.258062 16293 server_base.cc:1034] running on GCE node
I20260924 04:11:44.258837 16293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:11:44.260051 16293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:11:44.261255 16293 hybrid_clock.cc:648] HybridClock initialized: now 1790223104261238 us; error 35 us; skew 500 ppm
Sep 24 04:11:44 dist-test-slave-00ld krb5kdc[15746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223104, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:11:44.264079 16293 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:11:44.265077 16293 webserver.cc:460] Webserver started at http://127.0.98.62:37799/ using document root <none> and password file <none>
I20260924 04:11:44.265637 16293 fs_manager.cc:359] Metadata directory not provided
I20260924 04:11:44.265707 16293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:11:44.269593 16293 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:11:44.271781 16309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:11:44.273201 16293 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 04:11:44.273336 16293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/master-0/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/master-0/wal
uuid: "4b8cbc143cac4cd8bc3e990221c4f28e"
format_stamp: "Formatted at 2026-09-24 04:11:23 on dist-test-slave-00ld"
server_key: "e6e20e1cb40c8368cc2c1055d97fae4e"
server_key_iv: "a1aaffcaf6851dba7712560290aec965"
server_key_version: "encryptionkey@0"
I20260924 04:11:44.273945 16293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:11:44.301442 16293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:11:44.304925 16293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:11:44.305089 16293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:11:44.312214 16293 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37025
I20260924 04:11:44.312503 16361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37025 every 8 connection(s)
I20260924 04:11:44.313365 16293 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:11:44.313875 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 16293
I20260924 04:11:44.314424 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:42515
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:37025
--webserver_port=41211
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:11:44.318003 16362 sys_catalog.cc:263] Verifying existing consensus state
I20260924 04:11:44.319643 16362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b8cbc143cac4cd8bc3e990221c4f28e: Bootstrap starting.
I20260924 04:11:44.337033 16362 log.cc:824] T 00000000000000000000000000000000 P 4b8cbc143cac4cd8bc3e990221c4f28e: Log is configured to *not* fsync() on all Append() calls
I20260924 04:11:44.345718 16362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b8cbc143cac4cd8bc3e990221c4f28e: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 04:11:44.346123 16362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b8cbc143cac4cd8bc3e990221c4f28e: Bootstrap complete.
I20260924 04:11:44.350239 16362 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b8cbc143cac4cd8bc3e990221c4f28e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b8cbc143cac4cd8bc3e990221c4f28e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37025 } }
I20260924 04:11:44.350641 16362 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b8cbc143cac4cd8bc3e990221c4f28e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b8cbc143cac4cd8bc3e990221c4f28e, State: Initialized, Role: FOLLOWER
I20260924 04:11:44.351166 16362 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b8cbc143cac4cd8bc3e990221c4f28e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b8cbc143cac4cd8bc3e990221c4f28e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37025 } }
I20260924 04:11:44.351320 16362 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b8cbc143cac4cd8bc3e990221c4f28e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:11:44.351413 16362 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b8cbc143cac4cd8bc3e990221c4f28e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:11:44.351526 16362 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b8cbc143cac4cd8bc3e990221c4f28e [term 1 FOLLOWER]: Advancing to term 2
I20260924 04:11:44.353453 16362 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b8cbc143cac4cd8bc3e990221c4f28e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b8cbc143cac4cd8bc3e990221c4f28e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37025 } }
I20260924 04:11:44.353777 16362 leader_election.cc:302] T 00000000000000000000000000000000 P 4b8cbc143cac4cd8bc3e990221c4f28e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b8cbc143cac4cd8bc3e990221c4f28e; no voters:
I20260924 04:11:44.354076 16362 leader_election.cc:288] T 00000000000000000000000000000000 P 4b8cbc143cac4cd8bc3e990221c4f28e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 04:11:44.354215 16366 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b8cbc143cac4cd8bc3e990221c4f28e [term 2 FOLLOWER]: Leader election won for term 2
I20260924 04:11:44.354499 16366 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b8cbc143cac4cd8bc3e990221c4f28e [term 2 LEADER]: Becoming Leader. State: Replica: 4b8cbc143cac4cd8bc3e990221c4f28e, State: Running, Role: LEADER
I20260924 04:11:44.354791 16366 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b8cbc143cac4cd8bc3e990221c4f28e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b8cbc143cac4cd8bc3e990221c4f28e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37025 } }
I20260924 04:11:44.355293 16362 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b8cbc143cac4cd8bc3e990221c4f28e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:11:44.355715 16368 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b8cbc143cac4cd8bc3e990221c4f28e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b8cbc143cac4cd8bc3e990221c4f28e. Latest consensus state: current_term: 2 leader_uuid: "4b8cbc143cac4cd8bc3e990221c4f28e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b8cbc143cac4cd8bc3e990221c4f28e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37025 } } }
I20260924 04:11:44.355826 16368 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b8cbc143cac4cd8bc3e990221c4f28e [sys.catalog]: This master's current role is: LEADER
I20260924 04:11:44.356194 16367 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b8cbc143cac4cd8bc3e990221c4f28e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "4b8cbc143cac4cd8bc3e990221c4f28e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b8cbc143cac4cd8bc3e990221c4f28e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37025 } } }
I20260924 04:11:44.356324 16367 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b8cbc143cac4cd8bc3e990221c4f28e [sys.catalog]: This master's current role is: LEADER
I20260924 04:11:44.356618 16370 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:11:44.361681 16370 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:11:44.362277 16370 catalog_manager.cc:1259] Loaded cluster ID: 5cd8e9d1e20a4497b23d8cb482326859
I20260924 04:11:44.362386 16370 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:11:44.365130 16370 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:11:44.365950 16370 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4b8cbc143cac4cd8bc3e990221c4f28e: Loaded TSK: 0
I20260924 04:11:44.381842 16370 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4b8cbc143cac4cd8bc3e990221c4f28e: Generated new TSK 1
I20260924 04:11:44.382129 16370 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 04:11:44.431202 16364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:11:44.431481 16364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:11:44.431526 16364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:11:44.435632 16364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:11:44.435694 16364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:11:44.435793 16364 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:11:44.441236 16364 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:42515
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=41211
--tserver_master_addrs=127.0.98.62:37025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16364
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:11:44.442406 16364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:11:44.443504 16364 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:11:44.453373 16389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:11:44.453445 16364 server_base.cc:1034] running on GCE node
W20260924 04:11:44.453397 16392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:11:44.453646 16390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:11:44.454113 16364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:11:44.454849 16364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:11:44.456033 16364 hybrid_clock.cc:648] HybridClock initialized: now 1790223104456006 us; error 48 us; skew 500 ppm
Sep 24 04:11:44 dist-test-slave-00ld krb5kdc[15746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223104, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:11:44.459105 16364 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:11:44.460107 16364 webserver.cc:460] Webserver started at http://127.0.98.1:41211/ using document root <none> and password file <none>
I20260924 04:11:44.460772 16364 fs_manager.cc:359] Metadata directory not provided
I20260924 04:11:44.460853 16364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:11:44.465139 16364 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:11:44.467567 16399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:11:44.469007 16364 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:11:44.469203 16364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-0/wal
uuid: "29fe56f4efe74850870cd033fdea81d3"
format_stamp: "Formatted at 2026-09-24 04:11:23 on dist-test-slave-00ld"
server_key: "a8a753b5000d70b1e941547fbea391b0"
server_key_iv: "ff1443f5b70c7947ef0f1f9a3e1a936b"
server_key_version: "encryptionkey@0"
I20260924 04:11:44.469805 16364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:11:44.498677 16364 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:11:44.502206 16364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:11:44.502414 16364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:11:44.503075 16364 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:11:44.504204 16406 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 04:11:44.508306 16364 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 04:11:44.508369 16364 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.001s sys 0.000s
I20260924 04:11:44.508417 16364 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 04:11:44.510145 16364 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 04:11:44.510205 16364 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260924 04:11:44.510457 16406 tablet_bootstrap.cc:492] T 1ef81ae30bd7496787327eec67f0a089 P 29fe56f4efe74850870cd033fdea81d3: Bootstrap starting.
I20260924 04:11:44.521250 16364 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42515
I20260924 04:11:44.521313 16513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42515 every 8 connection(s)
I20260924 04:11:44.522523 16364 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:11:44.530900 16406 log.cc:824] T 1ef81ae30bd7496787327eec67f0a089 P 29fe56f4efe74850870cd033fdea81d3: Log is configured to *not* fsync() on all Append() calls
I20260924 04:11:44.531239 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 16364
I20260924 04:11:44.531800 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:38987
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:37025
--webserver_port=36121
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 04:11:44 dist-test-slave-00ld krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223104, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:11:44.539683 16406 tablet_bootstrap.cc:492] T 1ef81ae30bd7496787327eec67f0a089 P 29fe56f4efe74850870cd033fdea81d3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 04:11:44.539783 16517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:44.524649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.1:55769)
0924 04:11:44.525224 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0924 04:11:44.525266 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 04:11:44.526099 (+ 833us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:11:44.527181 (+ 1082us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:11:44.527188 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:11:44.527620 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:11:44.528067 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:11:44.528081 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:11:44.529469 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:11:44.529472 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:11:44.530339 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:11:44.530348 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:11:44.531212 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:11:44.531217 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:11:44.531474 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:11:44.532148 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:11:44.532168 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:11:44.533844 (+ 1676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:11:44.536238 (+ 2394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:11:44.536243 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:11:44.536256 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:11:44.536584 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:11:44.537069 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:11:44.537072 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:11:44.537074 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:11:44.537244 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:11:44.538023 (+ 779us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:11:44.538029 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:11:44.538315 (+ 286us) client_negotiation.cc:769] Sending connection context
0924 04:11:44.538504 (+ 189us) client_negotiation.cc:240] Negotiation successful
0924 04:11:44.538838 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":143,"threads_started":1}
I20260924 04:11:44.540009 16516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:44.524628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55769 (local address 127.0.98.62:37025)
0924 04:11:44.525033 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0924 04:11:44.525054 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0924 04:11:44.525391 (+ 337us) server_negotiation.cc:407] Connection header received
0924 04:11:44.526510 (+ 1119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:11:44.526520 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:11:44.526810 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:11:44.527144 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:11:44.528252 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:44.529300 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:44.530481 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:44.530990 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:44.531175 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:11:44.534052 (+ 2877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:11:44.534057 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:11:44.534070 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:11:44.534098 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:11:44.536096 (+ 1998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:11:44.536736 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:11:44.536744 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:11:44.536753 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:11:44.536941 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:11:44.537358 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:11:44.537364 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:11:44.537368 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:11:44.537886 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:11:44.538041 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 04:11:44.538676 (+ 635us) server_negotiation.cc:299] Negotiation successful
0924 04:11:44.539022 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":169,"threads_started":1}
I20260924 04:11:44.540130 16406 tablet_bootstrap.cc:492] T 1ef81ae30bd7496787327eec67f0a089 P 29fe56f4efe74850870cd033fdea81d3: Bootstrap complete.
I20260924 04:11:44.541386 16406 ts_tablet_manager.cc:1397] T 1ef81ae30bd7496787327eec67f0a089 P 29fe56f4efe74850870cd033fdea81d3: Time spent bootstrapping tablet: real 0.031s user 0.019s sys 0.011s
I20260924 04:11:44.542565 16514 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37025
I20260924 04:11:44.542830 16514 heartbeater.cc:464] Registering TS with master...
I20260924 04:11:44.543468 16514 heartbeater.cc:511] Master 127.0.98.62:37025 requested a full tablet report, sending...
I20260924 04:11:44.544236 16406 raft_consensus.cc:348] T 1ef81ae30bd7496787327eec67f0a089 P 29fe56f4efe74850870cd033fdea81d3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59864430ab9443eb82dc451f4a293582" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42569 } } peers { permanent_uuid: "b30b39da74e84f2a8af94206e4a2e8e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38987 } } peers { permanent_uuid: "29fe56f4efe74850870cd033fdea81d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42515 } }
I20260924 04:11:44.544690 16406 raft_consensus.cc:729] T 1ef81ae30bd7496787327eec67f0a089 P 29fe56f4efe74850870cd033fdea81d3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29fe56f4efe74850870cd033fdea81d3, State: Initialized, Role: FOLLOWER
I20260924 04:11:44.545225 16406 consensus_queue.cc:260] T 1ef81ae30bd7496787327eec67f0a089 P 29fe56f4efe74850870cd033fdea81d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59864430ab9443eb82dc451f4a293582" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42569 } } peers { permanent_uuid: "b30b39da74e84f2a8af94206e4a2e8e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38987 } } peers { permanent_uuid: "29fe56f4efe74850870cd033fdea81d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42515 } }
I20260924 04:11:44.546268 16406 ts_tablet_manager.cc:1428] T 1ef81ae30bd7496787327eec67f0a089 P 29fe56f4efe74850870cd033fdea81d3: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260924 04:11:44.546509 16326 ts_manager.cc:194] Registered new tserver with Master: 29fe56f4efe74850870cd033fdea81d3 (127.0.98.1:42515)
I20260924 04:11:44.550016 16326 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:55769
I20260924 04:11:44.552841 16514 heartbeater.cc:503] Master 127.0.98.62:37025 was elected leader, sending a full tablet report...
Sep 24 04:11:44 dist-test-slave-00ld krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223104, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 04:11:44.560525 16522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:44.549530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53388 (local address 127.0.98.1:42515)
0924 04:11:44.549927 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0924 04:11:44.549939 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 04:11:44.550367 (+ 428us) server_negotiation.cc:407] Connection header received
0924 04:11:44.550553 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:11:44.550559 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:11:44.550759 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:11:44.550890 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:11:44.551600 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:44.552317 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:44.553333 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:44.553858 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:44.553966 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:11:44.556747 (+ 2781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:11:44.556757 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:11:44.556787 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:11:44.556815 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:11:44.558415 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:11:44.558987 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:11:44.558993 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:11:44.558994 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:11:44.559081 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:11:44.559412 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:11:44.559415 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:11:44.559417 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:11:44.559719 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:11:44.559842 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 04:11:44.560249 (+ 407us) server_negotiation.cc:299] Negotiation successful
0924 04:11:44.560397 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":112,"threads_started":1}
I20260924 04:11:44.560525 16521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:44.549427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42515 (local address 127.0.0.1:53388)
0924 04:11:44.550213 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0924 04:11:44.550258 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 04:11:44.550401 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:11:44.550959 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:11:44.550965 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:11:44.551356 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:11:44.551463 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:11:44.551469 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:11:44.552437 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:11:44.552441 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:11:44.553212 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:11:44.553221 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:11:44.553973 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:11:44.553990 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:11:44.554122 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:11:44.554788 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:11:44.554803 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:11:44.556552 (+ 1749us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:11:44.558540 (+ 1988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:11:44.558551 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:11:44.558558 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:11:44.558854 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:11:44.559193 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:11:44.559197 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:11:44.559199 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:11:44.559294 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:11:44.559851 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:11:44.559857 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:11:44.559989 (+ 132us) client_negotiation.cc:769] Sending connection context
0924 04:11:44.560144 (+ 155us) client_negotiation.cc:240] Negotiation successful
0924 04:11:44.560348 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":107,"threads_started":1}
I20260924 04:11:44.561681 16448 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ef81ae30bd7496787327eec67f0a089 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:11:24 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53388
I20260924 04:11:44.562719 16523 tablet_replica.cc:330] stopping tablet replica
I20260924 04:11:44.563105 16523 raft_consensus.cc:2229] T 1ef81ae30bd7496787327eec67f0a089 P 29fe56f4efe74850870cd033fdea81d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:11:44.563380 16523 raft_consensus.cc:2258] T 1ef81ae30bd7496787327eec67f0a089 P 29fe56f4efe74850870cd033fdea81d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:11:44.564397 16523 ts_tablet_manager.cc:1905] T 1ef81ae30bd7496787327eec67f0a089 P 29fe56f4efe74850870cd033fdea81d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:11:44.568255 16523 ts_tablet_manager.cc:1918] T 1ef81ae30bd7496787327eec67f0a089 P 29fe56f4efe74850870cd033fdea81d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:11:44.568341 16523 log.cc:1196] T 1ef81ae30bd7496787327eec67f0a089 P 29fe56f4efe74850870cd033fdea81d3: Deleting WAL directory at /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-0/wal/wals/1ef81ae30bd7496787327eec67f0a089
I20260924 04:11:44.568629 16523 ts_tablet_manager.cc:1939] T 1ef81ae30bd7496787327eec67f0a089 P 29fe56f4efe74850870cd033fdea81d3: Deleting consensus metadata
I20260924 04:11:44.570506 16313 catalog_manager.cc:4841] TS 29fe56f4efe74850870cd033fdea81d3 (127.0.98.1:42515): tablet 1ef81ae30bd7496787327eec67f0a089 (table test-table [id=d1007511c3904fbab77fd65e4572c0a1]) successfully deleted
W20260924 04:11:44.650238 16519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:11:44.650522 16519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:11:44.650601 16519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:11:44.654695 16519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:11:44.654776 16519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:11:44.654865 16519 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:11:44.660503 16519 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:38987
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=36121
--tserver_master_addrs=127.0.98.62:37025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16519
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:11:44.661705 16519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:11:44.662818 16519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:11:44.672966 16531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:11:44.672977 16528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:11:44.673113 16519 server_base.cc:1034] running on GCE node
W20260924 04:11:44.672984 16529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:11:44.673782 16519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:11:44.674630 16519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:11:44.675819 16519 hybrid_clock.cc:648] HybridClock initialized: now 1790223104675810 us; error 33 us; skew 500 ppm
Sep 24 04:11:44 dist-test-slave-00ld krb5kdc[15746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223104, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:11:44.678751 16519 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:11:44.679612 16519 webserver.cc:460] Webserver started at http://127.0.98.2:36121/ using document root <none> and password file <none>
I20260924 04:11:44.680145 16519 fs_manager.cc:359] Metadata directory not provided
I20260924 04:11:44.680214 16519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:11:44.684305 16519 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:11:44.686551 16538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:11:44.687938 16519 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 04:11:44.688148 16519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-1/wal
uuid: "b30b39da74e84f2a8af94206e4a2e8e2"
format_stamp: "Formatted at 2026-09-24 04:11:23 on dist-test-slave-00ld"
server_key: "26eeade390c1ca46f79b67a27c04fa87"
server_key_iv: "edf98eef0f41d7c98248f520dee3a3b8"
server_key_version: "encryptionkey@0"
I20260924 04:11:44.688840 16519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:11:44.701429 16519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:11:44.704444 16519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:11:44.704651 16519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:11:44.705286 16519 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:11:44.706074 16519 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:11:44.706120 16519 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:11:44.706179 16519 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:11:44.706211 16519 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:11:44.715051 16519 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38987
I20260924 04:11:44.715085 16651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38987 every 8 connection(s)
I20260924 04:11:44.716044 16519 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:11:44.719326 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 16519
I20260924 04:11:44.719753 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:42569
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:37025
--webserver_port=39903
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 04:11:44 dist-test-slave-00ld krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223104, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:11:44.731813 16516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:44.717890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44169 (local address 127.0.98.62:37025)
0924 04:11:44.718026 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 04:11:44.718030 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:11:44.718860 (+ 830us) server_negotiation.cc:407] Connection header received
0924 04:11:44.720299 (+ 1439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:11:44.720302 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:11:44.720362 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:11:44.720470 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:11:44.722140 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:44.722703 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:44.724556 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:44.724839 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:44.724922 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:11:44.727783 (+ 2861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:11:44.727790 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:11:44.727796 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:11:44.727834 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:11:44.729642 (+ 1808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:11:44.730176 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:11:44.730181 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:11:44.730185 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:11:44.730285 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:11:44.730625 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:11:44.730631 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:11:44.730636 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:11:44.730908 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:11:44.731024 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 04:11:44.731499 (+ 475us) server_negotiation.cc:299] Negotiation successful
0924 04:11:44.731640 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 04:11:44.732565 16654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:44.718281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.2:44169)
0924 04:11:44.718706 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0924 04:11:44.718748 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 04:11:44.720073 (+ 1325us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:11:44.720652 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:11:44.720659 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:11:44.721254 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:11:44.721962 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:11:44.721984 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:11:44.722861 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:11:44.722867 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:11:44.724431 (+ 1564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:11:44.724446 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:11:44.725011 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:11:44.725017 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:11:44.725256 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:11:44.725915 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:11:44.725941 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:11:44.727611 (+ 1670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:11:44.729769 (+ 2158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:11:44.729774 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:11:44.729786 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:11:44.730073 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:11:44.730390 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:11:44.730392 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:11:44.730393 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:11:44.730523 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:11:44.731033 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:11:44.731038 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:11:44.731267 (+ 229us) client_negotiation.cc:769] Sending connection context
0924 04:11:44.731424 (+ 157us) client_negotiation.cc:240] Negotiation successful
0924 04:11:44.731674 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":114,"threads_started":1}
I20260924 04:11:44.733783 16652 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37025
I20260924 04:11:44.734038 16652 heartbeater.cc:464] Registering TS with master...
I20260924 04:11:44.734503 16652 heartbeater.cc:511] Master 127.0.98.62:37025 requested a full tablet report, sending...
I20260924 04:11:44.735507 16326 ts_manager.cc:194] Registered new tserver with Master: b30b39da74e84f2a8af94206e4a2e8e2 (127.0.98.2:38987)
I20260924 04:11:44.736081 16326 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44169
W20260924 04:11:44.834139 16655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:11:44.834419 16655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:11:44.834462 16655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:11:44.838624 16655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:11:44.838690 16655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:11:44.838781 16655 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:11:44.844086 16655 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:42569
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=39903
--tserver_master_addrs=127.0.98.62:37025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16655
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:11:44.845268 16655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:11:44.846350 16655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:11:44.855890 16661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:11:44.855923 16663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:11:44.855942 16660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:11:44.856379 16655 server_base.cc:1034] running on GCE node
I20260924 04:11:44.856904 16655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:11:44.857602 16655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:11:44.858770 16655 hybrid_clock.cc:648] HybridClock initialized: now 1790223104858757 us; error 26 us; skew 500 ppm
Sep 24 04:11:44 dist-test-slave-00ld krb5kdc[15746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223104, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:11:44.861533 16655 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:11:44.862391 16655 webserver.cc:460] Webserver started at http://127.0.98.3:39903/ using document root <none> and password file <none>
I20260924 04:11:44.862929 16655 fs_manager.cc:359] Metadata directory not provided
I20260924 04:11:44.862998 16655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:11:44.866871 16655 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260924 04:11:44.869364 16670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:11:44.870796 16655 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 04:11:44.870940 16655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-2/wal
uuid: "59864430ab9443eb82dc451f4a293582"
format_stamp: "Formatted at 2026-09-24 04:11:23 on dist-test-slave-00ld"
server_key: "1203e1471249287047c88de5eb16b603"
server_key_iv: "67fc4e595c5546b149140b244bc26258"
server_key_version: "encryptionkey@0"
I20260924 04:11:44.871505 16655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:11:44.888486 16655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:11:44.891649 16655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:11:44.891908 16655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:11:44.892601 16655 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:11:44.893613 16655 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:11:44.893662 16655 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:11:44.893702 16655 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:11:44.893718 16655 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:11:44.902308 16655 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42569
I20260924 04:11:44.902331 16783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42569 every 8 connection(s)
I20260924 04:11:44.903327 16655 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:11:44.907015 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 16655
Sep 24 04:11:44 dist-test-slave-00ld krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223104, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:11:44.916626 16788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:44.907689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59512 (local address 127.0.98.62:37025)
0924 04:11:44.907931 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0924 04:11:44.907935 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:11:44.908201 (+ 266us) server_negotiation.cc:407] Connection header received
0924 04:11:44.908392 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:11:44.908394 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:11:44.908432 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:11:44.908526 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:11:44.909313 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:44.909944 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:44.910812 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:44.911067 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:44.911179 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:11:44.912636 (+ 1457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:11:44.912644 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:11:44.912647 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:11:44.912667 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:11:44.914617 (+ 1950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:11:44.915213 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:11:44.915216 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:11:44.915217 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:11:44.915295 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:11:44.915583 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:11:44.915586 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:11:44.915588 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:11:44.915815 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:11:44.916040 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0924 04:11:44.916282 (+ 242us) server_negotiation.cc:299] Negotiation successful
0924 04:11:44.916406 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":177,"thread_start_us":89,"threads_started":1}
I20260924 04:11:44.917590 16516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:44.905215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49595 (local address 127.0.98.62:37025)
0924 04:11:44.905361 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 04:11:44.905365 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:11:44.906304 (+ 939us) server_negotiation.cc:407] Connection header received
0924 04:11:44.907730 (+ 1426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:11:44.907734 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:11:44.907788 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:11:44.907899 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:11:44.909312 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:44.909944 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:44.911074 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:44.911287 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:44.911396 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:11:44.913839 (+ 2443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:11:44.913845 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:11:44.913851 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:11:44.913909 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:11:44.915442 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:11:44.916014 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:11:44.916018 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:11:44.916020 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:11:44.916111 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:11:44.916531 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:11:44.916535 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:11:44.916538 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:11:44.916801 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:11:44.916903 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 04:11:44.917354 (+ 451us) server_negotiation.cc:299] Negotiation successful
0924 04:11:44.917464 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":88,"server-negotiator.queue_time_us":59}
I20260924 04:11:44.918479 16786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:44.905660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.3:49595)
0924 04:11:44.906129 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0924 04:11:44.906172 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 04:11:44.907461 (+ 1289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:11:44.908086 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:11:44.908093 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:11:44.908440 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:11:44.909135 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:11:44.909161 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:11:44.910077 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:11:44.910080 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:11:44.910944 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:11:44.910953 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:11:44.911433 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:11:44.911437 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:11:44.911601 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:11:44.912032 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:11:44.912047 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:11:44.913670 (+ 1623us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:11:44.915564 (+ 1894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:11:44.915571 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:11:44.915585 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:11:44.915908 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:11:44.916229 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:11:44.916231 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:11:44.916233 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:11:44.916400 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:11:44.916917 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:11:44.916923 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:11:44.917152 (+ 229us) client_negotiation.cc:769] Sending connection context
0924 04:11:44.917305 (+ 153us) client_negotiation.cc:240] Negotiation successful
0924 04:11:44.917592 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":120,"threads_started":1}
I20260924 04:11:44.919569 16784 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37025
I20260924 04:11:44.919765 16784 heartbeater.cc:464] Registering TS with master...
I20260924 04:11:44.920182 16784 heartbeater.cc:511] Master 127.0.98.62:37025 requested a full tablet report, sending...
I20260924 04:11:44.921162 16326 ts_manager.cc:194] Registered new tserver with Master: 59864430ab9443eb82dc451f4a293582 (127.0.98.3:42569)
I20260924 04:11:44.921703 16326 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49595
I20260924 04:11:44.930208 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:11:44.934671 16516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:44.930782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59526 (local address 127.0.98.62:37025)
0924 04:11:44.930906 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 04:11:44.930909 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:11:44.931200 (+ 291us) server_negotiation.cc:407] Connection header received
0924 04:11:44.931409 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:11:44.931414 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:11:44.931571 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:11:44.931697 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:11:44.932044 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:44.932599 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:44.933604 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:44.933822 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:44.933908 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:11:44.934240 (+ 332us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:11:44.934373 (+ 133us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0924 04:11:44.934572 (+ 199us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59526: token expired
Metrics: {"server-negotiator.queue_time_us":53}
W20260924 04:11:44.934767 16516 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59526: token expired
W20260924 04:11:44.935113 16253 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:37025)
I20260924 04:11:44.942903 16516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:44.935547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59542 (local address 127.0.98.62:37025)
0924 04:11:44.935657 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0924 04:11:44.935663 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:11:44.935678 (+ 15us) server_negotiation.cc:407] Connection header received
0924 04:11:44.935712 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:11:44.935714 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:11:44.935755 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:11:44.935838 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:11:44.936483 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:44.937044 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:44.937818 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:44.938038 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:44.938155 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:11:44.939552 (+ 1397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:11:44.939556 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:11:44.939559 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:11:44.939591 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:11:44.941202 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:11:44.941672 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:11:44.941676 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:11:44.941678 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:11:44.941760 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:11:44.942053 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:11:44.942056 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:11:44.942058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:11:44.942242 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:11:44.942380 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0924 04:11:44.942618 (+ 238us) server_negotiation.cc:299] Negotiation successful
0924 04:11:44.942735 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 04:11:44.945510 16326 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59542:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:11:44.948732 16326 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:11:44.956590 16448 tablet_service.cc:1467] Processing CreateTablet for tablet 53d4138f8ea243c8832b53ebcbb148ab (DEFAULT_TABLE table=test-table [id=a7142ab17d4f497ea76cda3cd4c3e72d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:11:44.957314 16448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53d4138f8ea243c8832b53ebcbb148ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:11:44.960716 16521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:44.955502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42569 (local address 127.0.0.1:38692)
0924 04:11:44.955623 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0924 04:11:44.955640 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 04:11:44.955718 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:11:44.956863 (+ 1145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:11:44.956866 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:11:44.956883 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:11:44.956987 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:11:44.956997 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:11:44.958586 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:11:44.958592 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:11:44.959403 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:11:44.959412 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:11:44.960229 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:11:44.960233 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:11:44.960324 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:11:44.960370 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 04:11:44.960444 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 04:11:44.960558 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260924 04:11:44.960736 16793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:44.955894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38692 (local address 127.0.98.3:42569)
0924 04:11:44.956284 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0924 04:11:44.956299 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 04:11:44.956324 (+ 25us) server_negotiation.cc:407] Connection header received
0924 04:11:44.956410 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:11:44.956416 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:11:44.956664 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:11:44.957162 (+ 498us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:11:44.957359 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:44.958471 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:44.959539 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:44.960106 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:44.960228 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:11:44.960383 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 04:11:44.960472 (+ 89us) server_negotiation.cc:299] Negotiation successful
0924 04:11:44.960559 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":207,"threads_started":1}
I20260924 04:11:44.960959 16406 tablet_bootstrap.cc:492] T 53d4138f8ea243c8832b53ebcbb148ab P 29fe56f4efe74850870cd033fdea81d3: Bootstrap starting.
I20260924 04:11:44.961604 16794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:44.956439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38987 (local address 127.0.0.1:56122)
0924 04:11:44.956983 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0924 04:11:44.956997 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:11:44.957081 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:11:44.957626 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:11:44.957629 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:11:44.957648 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:11:44.957729 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:11:44.957735 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:11:44.959013 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:11:44.959016 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:11:44.960159 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:11:44.960170 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:11:44.961226 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:11:44.961229 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:11:44.961313 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:11:44.961333 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 04:11:44.961397 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 04:11:44.961466 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":102,"threads_started":1}
I20260924 04:11:44.961663 16795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:44.956635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56122 (local address 127.0.98.2:38987)
0924 04:11:44.957098 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0924 04:11:44.957108 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 04:11:44.957131 (+ 23us) server_negotiation.cc:407] Connection header received
0924 04:11:44.957270 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:11:44.957276 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:11:44.957465 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:11:44.957634 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:11:44.957813 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:44.958869 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:44.960303 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:44.961091 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:44.961227 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:11:44.961376 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 04:11:44.961427 (+ 51us) server_negotiation.cc:299] Negotiation successful
0924 04:11:44.961531 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":178,"threads_started":1}
I20260924 04:11:44.961707 16406 tablet_bootstrap.cc:654] T 53d4138f8ea243c8832b53ebcbb148ab P 29fe56f4efe74850870cd033fdea81d3: Neither blocks nor log segments found. Creating new log.
I20260924 04:11:44.963735 16406 tablet_bootstrap.cc:492] T 53d4138f8ea243c8832b53ebcbb148ab P 29fe56f4efe74850870cd033fdea81d3: No bootstrap required, opened a new log
I20260924 04:11:44.963822 16406 ts_tablet_manager.cc:1397] T 53d4138f8ea243c8832b53ebcbb148ab P 29fe56f4efe74850870cd033fdea81d3: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260924 04:11:44.964251 16406 raft_consensus.cc:348] T 53d4138f8ea243c8832b53ebcbb148ab P 29fe56f4efe74850870cd033fdea81d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59864430ab9443eb82dc451f4a293582" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42569 } } peers { permanent_uuid: "b30b39da74e84f2a8af94206e4a2e8e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38987 } } peers { permanent_uuid: "29fe56f4efe74850870cd033fdea81d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42515 } }
I20260924 04:11:44.964365 16406 raft_consensus.cc:374] T 53d4138f8ea243c8832b53ebcbb148ab P 29fe56f4efe74850870cd033fdea81d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:11:44.964402 16406 raft_consensus.cc:729] T 53d4138f8ea243c8832b53ebcbb148ab P 29fe56f4efe74850870cd033fdea81d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29fe56f4efe74850870cd033fdea81d3, State: Initialized, Role: FOLLOWER
I20260924 04:11:44.964538 16406 consensus_queue.cc:260] T 53d4138f8ea243c8832b53ebcbb148ab P 29fe56f4efe74850870cd033fdea81d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59864430ab9443eb82dc451f4a293582" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42569 } } peers { permanent_uuid: "b30b39da74e84f2a8af94206e4a2e8e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38987 } } peers { permanent_uuid: "29fe56f4efe74850870cd033fdea81d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42515 } }
I20260924 04:11:44.964805 16406 ts_tablet_manager.cc:1428] T 53d4138f8ea243c8832b53ebcbb148ab P 29fe56f4efe74850870cd033fdea81d3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:11:44.964986 16586 tablet_service.cc:1467] Processing CreateTablet for tablet 53d4138f8ea243c8832b53ebcbb148ab (DEFAULT_TABLE table=test-table [id=a7142ab17d4f497ea76cda3cd4c3e72d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:11:44.965579 16718 tablet_service.cc:1467] Processing CreateTablet for tablet 53d4138f8ea243c8832b53ebcbb148ab (DEFAULT_TABLE table=test-table [id=a7142ab17d4f497ea76cda3cd4c3e72d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:11:44.966204 16586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53d4138f8ea243c8832b53ebcbb148ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:11:44.966594 16718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53d4138f8ea243c8832b53ebcbb148ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:11:44.972968 16797 tablet_bootstrap.cc:492] T 53d4138f8ea243c8832b53ebcbb148ab P 59864430ab9443eb82dc451f4a293582: Bootstrap starting.
I20260924 04:11:44.972968 16798 tablet_bootstrap.cc:492] T 53d4138f8ea243c8832b53ebcbb148ab P b30b39da74e84f2a8af94206e4a2e8e2: Bootstrap starting.
I20260924 04:11:44.975052 16797 tablet_bootstrap.cc:654] T 53d4138f8ea243c8832b53ebcbb148ab P 59864430ab9443eb82dc451f4a293582: Neither blocks nor log segments found. Creating new log.
I20260924 04:11:44.976047 16798 tablet_bootstrap.cc:654] T 53d4138f8ea243c8832b53ebcbb148ab P b30b39da74e84f2a8af94206e4a2e8e2: Neither blocks nor log segments found. Creating new log.
I20260924 04:11:44.976413 16797 log.cc:824] T 53d4138f8ea243c8832b53ebcbb148ab P 59864430ab9443eb82dc451f4a293582: Log is configured to *not* fsync() on all Append() calls
I20260924 04:11:44.976913 16798 log.cc:824] T 53d4138f8ea243c8832b53ebcbb148ab P b30b39da74e84f2a8af94206e4a2e8e2: Log is configured to *not* fsync() on all Append() calls
I20260924 04:11:44.978571 16797 tablet_bootstrap.cc:492] T 53d4138f8ea243c8832b53ebcbb148ab P 59864430ab9443eb82dc451f4a293582: No bootstrap required, opened a new log
I20260924 04:11:44.978691 16798 tablet_bootstrap.cc:492] T 53d4138f8ea243c8832b53ebcbb148ab P b30b39da74e84f2a8af94206e4a2e8e2: No bootstrap required, opened a new log
I20260924 04:11:44.978808 16797 ts_tablet_manager.cc:1397] T 53d4138f8ea243c8832b53ebcbb148ab P 59864430ab9443eb82dc451f4a293582: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 04:11:44.978876 16798 ts_tablet_manager.cc:1397] T 53d4138f8ea243c8832b53ebcbb148ab P b30b39da74e84f2a8af94206e4a2e8e2: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260924 04:11:44.982105 16797 raft_consensus.cc:348] T 53d4138f8ea243c8832b53ebcbb148ab P 59864430ab9443eb82dc451f4a293582 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59864430ab9443eb82dc451f4a293582" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42569 } } peers { permanent_uuid: "b30b39da74e84f2a8af94206e4a2e8e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38987 } } peers { permanent_uuid: "29fe56f4efe74850870cd033fdea81d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42515 } }
I20260924 04:11:44.982108 16798 raft_consensus.cc:348] T 53d4138f8ea243c8832b53ebcbb148ab P b30b39da74e84f2a8af94206e4a2e8e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59864430ab9443eb82dc451f4a293582" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42569 } } peers { permanent_uuid: "b30b39da74e84f2a8af94206e4a2e8e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38987 } } peers { permanent_uuid: "29fe56f4efe74850870cd033fdea81d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42515 } }
I20260924 04:11:44.982369 16797 raft_consensus.cc:374] T 53d4138f8ea243c8832b53ebcbb148ab P 59864430ab9443eb82dc451f4a293582 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:11:44.982364 16798 raft_consensus.cc:374] T 53d4138f8ea243c8832b53ebcbb148ab P b30b39da74e84f2a8af94206e4a2e8e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:11:44.982430 16797 raft_consensus.cc:729] T 53d4138f8ea243c8832b53ebcbb148ab P 59864430ab9443eb82dc451f4a293582 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59864430ab9443eb82dc451f4a293582, State: Initialized, Role: FOLLOWER
I20260924 04:11:44.982440 16798 raft_consensus.cc:729] T 53d4138f8ea243c8832b53ebcbb148ab P b30b39da74e84f2a8af94206e4a2e8e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b30b39da74e84f2a8af94206e4a2e8e2, State: Initialized, Role: FOLLOWER
I20260924 04:11:44.982836 16798 consensus_queue.cc:260] T 53d4138f8ea243c8832b53ebcbb148ab P b30b39da74e84f2a8af94206e4a2e8e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59864430ab9443eb82dc451f4a293582" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42569 } } peers { permanent_uuid: "b30b39da74e84f2a8af94206e4a2e8e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38987 } } peers { permanent_uuid: "29fe56f4efe74850870cd033fdea81d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42515 } }
I20260924 04:11:44.982867 16797 consensus_queue.cc:260] T 53d4138f8ea243c8832b53ebcbb148ab P 59864430ab9443eb82dc451f4a293582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59864430ab9443eb82dc451f4a293582" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42569 } } peers { permanent_uuid: "b30b39da74e84f2a8af94206e4a2e8e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38987 } } peers { permanent_uuid: "29fe56f4efe74850870cd033fdea81d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42515 } }
I20260924 04:11:44.983477 16784 heartbeater.cc:503] Master 127.0.98.62:37025 was elected leader, sending a full tablet report...
I20260924 04:11:44.983477 16652 heartbeater.cc:503] Master 127.0.98.62:37025 was elected leader, sending a full tablet report...
I20260924 04:11:44.983855 16797 ts_tablet_manager.cc:1428] T 53d4138f8ea243c8832b53ebcbb148ab P 59864430ab9443eb82dc451f4a293582: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260924 04:11:44.983862 16798 ts_tablet_manager.cc:1428] T 53d4138f8ea243c8832b53ebcbb148ab P b30b39da74e84f2a8af94206e4a2e8e2: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
W20260924 04:11:45.023732 16515 tablet.cc:2369] T 53d4138f8ea243c8832b53ebcbb148ab P 29fe56f4efe74850870cd033fdea81d3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:11:45.088474 16802 raft_consensus.cc:482] T 53d4138f8ea243c8832b53ebcbb148ab P 59864430ab9443eb82dc451f4a293582 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:11:45.088734 16802 raft_consensus.cc:504] T 53d4138f8ea243c8832b53ebcbb148ab P 59864430ab9443eb82dc451f4a293582 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59864430ab9443eb82dc451f4a293582" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42569 } } peers { permanent_uuid: "b30b39da74e84f2a8af94206e4a2e8e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38987 } } peers { permanent_uuid: "29fe56f4efe74850870cd033fdea81d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42515 } }
I20260924 04:11:45.089874 16802 leader_election.cc:288] T 53d4138f8ea243c8832b53ebcbb148ab P 59864430ab9443eb82dc451f4a293582 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b30b39da74e84f2a8af94206e4a2e8e2 (127.0.98.2:38987), 29fe56f4efe74850870cd033fdea81d3 (127.0.98.1:42515)
I20260924 04:11:45.094003 16803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:45.090098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38987 (local address 127.0.98.3:46733)
0924 04:11:45.090454 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0924 04:11:45.090467 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:11:45.090596 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:11:45.090857 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:11:45.090860 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:11:45.090875 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:11:45.091008 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:11:45.091013 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:11:45.091727 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:11:45.091732 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:11:45.092818 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:11:45.092827 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:11:45.093523 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:11:45.093528 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:11:45.093673 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:11:45.093697 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 04:11:45.093770 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 04:11:45.093849 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"mutex_wait_us":98,"spinlock_wait_cycles":24832}
I20260924 04:11:45.094003 16795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:45.090202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46733 (local address 127.0.98.2:38987)
0924 04:11:45.090339 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 04:11:45.090346 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 04:11:45.090554 (+ 208us) server_negotiation.cc:407] Connection header received
0924 04:11:45.090705 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:11:45.090709 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:11:45.090759 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:11:45.090838 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:11:45.091114 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:45.091618 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:45.092942 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:45.093371 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:45.093514 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:11:45.093589 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 04:11:45.093801 (+ 212us) server_negotiation.cc:299] Negotiation successful
0924 04:11:45.093885 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 04:11:45.094097 16804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:45.090239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33991 (local address 127.0.98.1:42515)
0924 04:11:45.090487 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0924 04:11:45.090493 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:11:45.090513 (+ 20us) server_negotiation.cc:407] Connection header received
0924 04:11:45.090709 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:11:45.090712 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:11:45.090766 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:11:45.090871 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:11:45.091167 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:45.091662 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:45.092921 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:45.093420 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:45.093604 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:11:45.093783 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0924 04:11:45.093890 (+ 107us) server_negotiation.cc:299] Negotiation successful
0924 04:11:45.093958 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":94,"threads_started":1}
I20260924 04:11:45.094501 16801 raft_consensus.cc:482] T 53d4138f8ea243c8832b53ebcbb148ab P b30b39da74e84f2a8af94206e4a2e8e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:11:45.094700 16801 raft_consensus.cc:504] T 53d4138f8ea243c8832b53ebcbb148ab P b30b39da74e84f2a8af94206e4a2e8e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59864430ab9443eb82dc451f4a293582" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42569 } } peers { permanent_uuid: "b30b39da74e84f2a8af94206e4a2e8e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38987 } } peers { permanent_uuid: "29fe56f4efe74850870cd033fdea81d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42515 } }
I20260924 04:11:45.095031 16606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53d4138f8ea243c8832b53ebcbb148ab" candidate_uuid: "59864430ab9443eb82dc451f4a293582" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b30b39da74e84f2a8af94206e4a2e8e2" is_pre_election: true
I20260924 04:11:45.095252 16606 raft_consensus.cc:2454] T 53d4138f8ea243c8832b53ebcbb148ab P b30b39da74e84f2a8af94206e4a2e8e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59864430ab9443eb82dc451f4a293582 in term 0.
I20260924 04:11:45.095708 16673 leader_election.cc:302] T 53d4138f8ea243c8832b53ebcbb148ab P 59864430ab9443eb82dc451f4a293582 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59864430ab9443eb82dc451f4a293582, b30b39da74e84f2a8af94206e4a2e8e2; no voters:
I20260924 04:11:45.095948 16801 leader_election.cc:288] T 53d4138f8ea243c8832b53ebcbb148ab P b30b39da74e84f2a8af94206e4a2e8e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59864430ab9443eb82dc451f4a293582 (127.0.98.3:42569), 29fe56f4efe74850870cd033fdea81d3 (127.0.98.1:42515)
I20260924 04:11:45.096108 16802 raft_consensus.cc:2781] T 53d4138f8ea243c8832b53ebcbb148ab P 59864430ab9443eb82dc451f4a293582 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:11:45.096194 16802 raft_consensus.cc:482] T 53d4138f8ea243c8832b53ebcbb148ab P 59864430ab9443eb82dc451f4a293582 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:11:45.096257 16802 raft_consensus.cc:3037] T 53d4138f8ea243c8832b53ebcbb148ab P 59864430ab9443eb82dc451f4a293582 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:11:45.097711 16802 raft_consensus.cc:504] T 53d4138f8ea243c8832b53ebcbb148ab P 59864430ab9443eb82dc451f4a293582 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59864430ab9443eb82dc451f4a293582" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42569 } } peers { permanent_uuid: "b30b39da74e84f2a8af94206e4a2e8e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38987 } } peers { permanent_uuid: "29fe56f4efe74850870cd033fdea81d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42515 } }
I20260924 04:11:45.098073 16802 leader_election.cc:288] T 53d4138f8ea243c8832b53ebcbb148ab P 59864430ab9443eb82dc451f4a293582 [CANDIDATE]: Term 1 election: Requested vote from peers b30b39da74e84f2a8af94206e4a2e8e2 (127.0.98.2:38987), 29fe56f4efe74850870cd033fdea81d3 (127.0.98.1:42515)
I20260924 04:11:45.094058 16786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:45.090130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42515 (local address 127.0.98.3:33991)
0924 04:11:45.090458 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0924 04:11:45.090473 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:11:45.090577 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:11:45.090889 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:11:45.090891 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:11:45.090928 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:11:45.091034 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:11:45.091040 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:11:45.091798 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:11:45.091805 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:11:45.092803 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:11:45.092813 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:11:45.093597 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:11:45.093602 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:11:45.093763 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:11:45.093788 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 04:11:45.093863 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 04:11:45.093933 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"mutex_wait_us":65,"spinlock_wait_cycles":1408,"thread_start_us":142,"threads_started":1}
I20260924 04:11:45.098505 16606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53d4138f8ea243c8832b53ebcbb148ab" candidate_uuid: "59864430ab9443eb82dc451f4a293582" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b30b39da74e84f2a8af94206e4a2e8e2"
I20260924 04:11:45.098618 16606 raft_consensus.cc:3037] T 53d4138f8ea243c8832b53ebcbb148ab P b30b39da74e84f2a8af94206e4a2e8e2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:11:45.099848 16606 raft_consensus.cc:2454] T 53d4138f8ea243c8832b53ebcbb148ab P b30b39da74e84f2a8af94206e4a2e8e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59864430ab9443eb82dc451f4a293582 in term 1.
I20260924 04:11:45.100098 16793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:45.096474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60421 (local address 127.0.98.3:42569)
0924 04:11:45.096587 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0924 04:11:45.096593 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:11:45.096607 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:11:45.096648 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:11:45.096650 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:11:45.096724 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:11:45.096800 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:11:45.097064 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:45.097652 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:45.099125 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:45.099584 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:45.099699 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:11:45.099753 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 04:11:45.099927 (+ 174us) server_negotiation.cc:299] Negotiation successful
0924 04:11:45.099985 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 04:11:45.100112 16654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:45.096090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42569 (local address 127.0.98.2:60421)
0924 04:11:45.096304 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0924 04:11:45.096320 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:11:45.096406 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:11:45.096813 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:11:45.096816 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:11:45.096850 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:11:45.096953 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:11:45.096958 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:11:45.097759 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:11:45.097764 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:11:45.098997 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:11:45.099009 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:11:45.099699 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:11:45.099702 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:11:45.099796 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:11:45.099817 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 04:11:45.099913 (+ 96us) client_negotiation.cc:240] Negotiation successful
0924 04:11:45.099982 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":32,"spinlock_wait_cycles":10752}
I20260924 04:11:45.100225 16468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53d4138f8ea243c8832b53ebcbb148ab" candidate_uuid: "59864430ab9443eb82dc451f4a293582" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29fe56f4efe74850870cd033fdea81d3" is_pre_election: true
I20260924 04:11:45.100543 16805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:45.096181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42515 (local address 127.0.98.2:47687)
0924 04:11:45.096736 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0924 04:11:45.096749 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:11:45.096847 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:11:45.097109 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:11:45.097112 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:11:45.097129 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:11:45.097231 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:11:45.097236 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:11:45.098129 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:11:45.098132 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:11:45.099180 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:11:45.099190 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:11:45.100132 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:11:45.100135 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:11:45.100226 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:11:45.100257 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 04:11:45.100338 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 04:11:45.100403 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"mutex_wait_us":20,"spinlock_wait_cycles":4992,"thread_start_us":110,"threads_started":1}
I20260924 04:11:45.100505 16804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:45.096239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47687 (local address 127.0.98.1:42515)
0924 04:11:45.096351 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0924 04:11:45.096354 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:11:45.096926 (+ 572us) server_negotiation.cc:407] Connection header received
0924 04:11:45.096970 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:11:45.096973 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:11:45.097031 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:11:45.097101 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:11:45.097349 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:45.097983 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:45.099309 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:45.099915 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:45.100017 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:11:45.100074 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 04:11:45.100353 (+ 279us) server_negotiation.cc:299] Negotiation successful
0924 04:11:45.100424 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 04:11:45.100522 16468 raft_consensus.cc:2454] T 53d4138f8ea243c8832b53ebcbb148ab P 29fe56f4efe74850870cd033fdea81d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59864430ab9443eb82dc451f4a293582 in term 0.
I20260924 04:11:45.100629 16467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53d4138f8ea243c8832b53ebcbb148ab" candidate_uuid: "59864430ab9443eb82dc451f4a293582" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29fe56f4efe74850870cd033fdea81d3"
I20260924 04:11:45.100772 16467 raft_consensus.cc:3037] T 53d4138f8ea243c8832b53ebcbb148ab P 29fe56f4efe74850870cd033fdea81d3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:11:45.101687 16467 raft_consensus.cc:2454] T 53d4138f8ea243c8832b53ebcbb148ab P 29fe56f4efe74850870cd033fdea81d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59864430ab9443eb82dc451f4a293582 in term 1.
I20260924 04:11:45.101982 16674 leader_election.cc:302] T 53d4138f8ea243c8832b53ebcbb148ab P 59864430ab9443eb82dc451f4a293582 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29fe56f4efe74850870cd033fdea81d3, 59864430ab9443eb82dc451f4a293582; no voters:
I20260924 04:11:45.102042 16467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53d4138f8ea243c8832b53ebcbb148ab" candidate_uuid: "b30b39da74e84f2a8af94206e4a2e8e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29fe56f4efe74850870cd033fdea81d3" is_pre_election: true
I20260924 04:11:45.102160 16802 raft_consensus.cc:2781] T 53d4138f8ea243c8832b53ebcbb148ab P 59864430ab9443eb82dc451f4a293582 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:11:45.102232 16467 raft_consensus.cc:2379] T 53d4138f8ea243c8832b53ebcbb148ab P 29fe56f4efe74850870cd033fdea81d3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b30b39da74e84f2a8af94206e4a2e8e2 in current term 1: Already voted for candidate 59864430ab9443eb82dc451f4a293582 in this term.
I20260924 04:11:45.102361 16802 raft_consensus.cc:686] T 53d4138f8ea243c8832b53ebcbb148ab P 59864430ab9443eb82dc451f4a293582 [term 1 LEADER]: Becoming Leader. State: Replica: 59864430ab9443eb82dc451f4a293582, State: Running, Role: LEADER
I20260924 04:11:45.102735 16802 consensus_queue.cc:237] T 53d4138f8ea243c8832b53ebcbb148ab P 59864430ab9443eb82dc451f4a293582 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59864430ab9443eb82dc451f4a293582" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42569 } } peers { permanent_uuid: "b30b39da74e84f2a8af94206e4a2e8e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38987 } } peers { permanent_uuid: "29fe56f4efe74850870cd033fdea81d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42515 } }
I20260924 04:11:45.102744 16738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53d4138f8ea243c8832b53ebcbb148ab" candidate_uuid: "b30b39da74e84f2a8af94206e4a2e8e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59864430ab9443eb82dc451f4a293582" is_pre_election: true
I20260924 04:11:45.103410 16539 leader_election.cc:302] T 53d4138f8ea243c8832b53ebcbb148ab P b30b39da74e84f2a8af94206e4a2e8e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b30b39da74e84f2a8af94206e4a2e8e2; no voters: 29fe56f4efe74850870cd033fdea81d3, 59864430ab9443eb82dc451f4a293582
I20260924 04:11:45.103698 16801 raft_consensus.cc:2726] T 53d4138f8ea243c8832b53ebcbb148ab P b30b39da74e84f2a8af94206e4a2e8e2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 04:11:45.106081 16322 catalog_manager.cc:5495] T 53d4138f8ea243c8832b53ebcbb148ab P 59864430ab9443eb82dc451f4a293582 reported cstate change: term changed from 0 to 1, leader changed from <none> to 59864430ab9443eb82dc451f4a293582 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "59864430ab9443eb82dc451f4a293582" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59864430ab9443eb82dc451f4a293582" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b30b39da74e84f2a8af94206e4a2e8e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29fe56f4efe74850870cd033fdea81d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42515 } health_report { overall_health: UNKNOWN } } }
I20260924 04:11:45.117313 16793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:45.114160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38708 (local address 127.0.98.3:42569)
0924 04:11:45.114271 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0924 04:11:45.114275 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:11:45.114287 (+ 12us) server_negotiation.cc:407] Connection header received
0924 04:11:45.114421 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:11:45.114423 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:11:45.114470 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:11:45.114555 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:11:45.114752 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:45.115233 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:45.116017 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:11:45.116218 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:11:45.116290 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:11:45.116586 (+ 296us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:11:45.116985 (+ 399us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:11:45.117070 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 04:11:45.117176 (+ 106us) server_negotiation.cc:299] Negotiation successful
0924 04:11:45.117219 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 04:11:45.122884 16467 raft_consensus.cc:1261] T 53d4138f8ea243c8832b53ebcbb148ab P 29fe56f4efe74850870cd033fdea81d3 [term 1 FOLLOWER]: Refusing update from remote peer 59864430ab9443eb82dc451f4a293582: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:11:45.122923 16606 raft_consensus.cc:1261] T 53d4138f8ea243c8832b53ebcbb148ab P b30b39da74e84f2a8af94206e4a2e8e2 [term 1 FOLLOWER]: Refusing update from remote peer 59864430ab9443eb82dc451f4a293582: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:11:45.123549 16806 consensus_queue.cc:1035] T 53d4138f8ea243c8832b53ebcbb148ab P 59864430ab9443eb82dc451f4a293582 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29fe56f4efe74850870cd033fdea81d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:11:45.123735 16802 consensus_queue.cc:1035] T 53d4138f8ea243c8832b53ebcbb148ab P 59864430ab9443eb82dc451f4a293582 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b30b39da74e84f2a8af94206e4a2e8e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:11:45.130857 16810 mvcc.cc:205] Tried to move back new op lower bound from 7332753838576283648 to 7332753838504067072. Current Snapshot: MvccSnapshot[applied={T|T < 7332753838576283648}]
I20260924 04:11:45.131304 16811 mvcc.cc:205] Tried to move back new op lower bound from 7332753838576283648 to 7332753838504067072. Current Snapshot: MvccSnapshot[applied={T|T < 7332753838576283648}]
I20260924 04:11:45.138721 16322 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59542:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:11:45.138904 16322 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59542:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:11:45.141369 16322 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4b8cbc143cac4cd8bc3e990221c4f28e: Sending DeleteTablet for 3 replicas of tablet 53d4138f8ea243c8832b53ebcbb148ab
I20260924 04:11:45.142086 16718 tablet_service.cc:1514] Processing DeleteTablet for tablet 53d4138f8ea243c8832b53ebcbb148ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:11:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38692
I20260924 04:11:45.142174 16586 tablet_service.cc:1514] Processing DeleteTablet for tablet 53d4138f8ea243c8832b53ebcbb148ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:11:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56122
I20260924 04:11:45.142589 16818 tablet_replica.cc:330] T 53d4138f8ea243c8832b53ebcbb148ab P 59864430ab9443eb82dc451f4a293582: stopping tablet replica
I20260924 04:11:45.142590 16448 tablet_service.cc:1514] Processing DeleteTablet for tablet 53d4138f8ea243c8832b53ebcbb148ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:11:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53388
I20260924 04:11:45.142666 16819 tablet_replica.cc:330] T 53d4138f8ea243c8832b53ebcbb148ab P b30b39da74e84f2a8af94206e4a2e8e2: stopping tablet replica
I20260924 04:11:45.142979 16819 raft_consensus.cc:2229] T 53d4138f8ea243c8832b53ebcbb148ab P b30b39da74e84f2a8af94206e4a2e8e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:11:45.142979 16818 raft_consensus.cc:2229] T 53d4138f8ea243c8832b53ebcbb148ab P 59864430ab9443eb82dc451f4a293582 [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:11:45.143067 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 16364
I20260924 04:11:45.143309 16819 raft_consensus.cc:2258] T 53d4138f8ea243c8832b53ebcbb148ab P b30b39da74e84f2a8af94206e4a2e8e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:11:45.144063 16818 raft_consensus.cc:2258] T 53d4138f8ea243c8832b53ebcbb148ab P 59864430ab9443eb82dc451f4a293582 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:11:45.145328 16819 ts_tablet_manager.cc:1905] T 53d4138f8ea243c8832b53ebcbb148ab P b30b39da74e84f2a8af94206e4a2e8e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:11:45.146893 16818 ts_tablet_manager.cc:1905] T 53d4138f8ea243c8832b53ebcbb148ab P 59864430ab9443eb82dc451f4a293582: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:11:45.148313 16819 ts_tablet_manager.cc:1918] T 53d4138f8ea243c8832b53ebcbb148ab P b30b39da74e84f2a8af94206e4a2e8e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:11:45.148427 16819 log.cc:1196] T 53d4138f8ea243c8832b53ebcbb148ab P b30b39da74e84f2a8af94206e4a2e8e2: Deleting WAL directory at /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-1/wal/wals/53d4138f8ea243c8832b53ebcbb148ab
I20260924 04:11:45.149209 16819 ts_tablet_manager.cc:1939] T 53d4138f8ea243c8832b53ebcbb148ab P b30b39da74e84f2a8af94206e4a2e8e2: Deleting consensus metadata
I20260924 04:11:45.150107 16818 ts_tablet_manager.cc:1918] T 53d4138f8ea243c8832b53ebcbb148ab P 59864430ab9443eb82dc451f4a293582: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:11:45.150264 16818 log.cc:1196] T 53d4138f8ea243c8832b53ebcbb148ab P 59864430ab9443eb82dc451f4a293582: Deleting WAL directory at /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-2/wal/wals/53d4138f8ea243c8832b53ebcbb148ab
I20260924 04:11:45.150619 16312 catalog_manager.cc:4841] TS b30b39da74e84f2a8af94206e4a2e8e2 (127.0.98.2:38987): tablet 53d4138f8ea243c8832b53ebcbb148ab (table test-table [id=a7142ab17d4f497ea76cda3cd4c3e72d]) successfully deleted
I20260924 04:11:45.150707 16818 ts_tablet_manager.cc:1939] T 53d4138f8ea243c8832b53ebcbb148ab P 59864430ab9443eb82dc451f4a293582: Deleting consensus metadata
W20260924 04:11:45.151638 16313 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42515 (error 108)
I20260924 04:11:45.152120 16310 catalog_manager.cc:4841] TS 59864430ab9443eb82dc451f4a293582 (127.0.98.3:42569): tablet 53d4138f8ea243c8832b53ebcbb148ab (table test-table [id=a7142ab17d4f497ea76cda3cd4c3e72d]) successfully deleted
I20260924 04:11:45.152789 16794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:11:45.152492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42515 (local address 127.0.0.1:53400)
0924 04:11:45.152609 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0924 04:11:45.152700 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260924 04:11:45.153182 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 16519
W20260924 04:11:45.152992 16313 catalog_manager.cc:4584] TS 29fe56f4efe74850870cd033fdea81d3 (127.0.98.1:42515): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 53d4138f8ea243c8832b53ebcbb148ab: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42515: connect: Connection refused (error 111)
I20260924 04:11:45.160493 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 16655
I20260924 04:11:45.166519 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 16293
I20260924 04:12:05.173593 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37025
--webserver_interface=127.0.98.62
--webserver_port=37799
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37025
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:12:05.283078 16822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:12:05.283391 16822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:12:05.283468 16822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:12:05.287464 16822 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 04:12:05.287545 16822 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 04:12:05.287572 16822 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 04:12:05.287604 16822 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:12:05.287647 16822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:12:05.287686 16822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:12:05.287722 16822 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:12:05.287758 16822 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:12:05.293322 16822 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37025
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=37799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16822
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:12:05.294416 16822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:12:05.295449 16822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:12:05.304252 16828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:12:05.304301 16830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:12:05.304351 16827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:12:05.304416 16822 server_base.cc:1034] running on GCE node
I20260924 04:12:05.305035 16822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:12:05.306136 16822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:12:05.307307 16822 hybrid_clock.cc:648] HybridClock initialized: now 1790223125307290 us; error 45 us; skew 500 ppm
Sep 24 04:12:05 dist-test-slave-00ld krb5kdc[15746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223125, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:12:05.310104 16822 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:12:05.310920 16822 webserver.cc:460] Webserver started at http://127.0.98.62:37799/ using document root <none> and password file <none>
I20260924 04:12:05.311450 16822 fs_manager.cc:359] Metadata directory not provided
I20260924 04:12:05.311510 16822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:12:05.315233 16822 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.001s
I20260924 04:12:05.317358 16837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:12:05.318528 16822 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:12:05.318642 16822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/master-0/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/master-0/wal
uuid: "4b8cbc143cac4cd8bc3e990221c4f28e"
format_stamp: "Formatted at 2026-09-24 04:11:23 on dist-test-slave-00ld"
server_key: "e6e20e1cb40c8368cc2c1055d97fae4e"
server_key_iv: "a1aaffcaf6851dba7712560290aec965"
server_key_version: "encryptionkey@0"
I20260924 04:12:05.319242 16822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:12:05.336971 16822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:12:05.340370 16822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:12:05.340588 16822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:12:05.350289 16822 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37025
I20260924 04:12:05.350593 16889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37025 every 8 connection(s)
I20260924 04:12:05.351437 16822 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:12:05.355885 16890 sys_catalog.cc:263] Verifying existing consensus state
I20260924 04:12:05.357568 16890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b8cbc143cac4cd8bc3e990221c4f28e: Bootstrap starting.
I20260924 04:12:05.360105 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 16822
I20260924 04:12:05.360638 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:42515
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:37025
--webserver_port=41211
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:12:05.375481 16890 log.cc:824] T 00000000000000000000000000000000 P 4b8cbc143cac4cd8bc3e990221c4f28e: Log is configured to *not* fsync() on all Append() calls
I20260924 04:12:05.386646 16890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b8cbc143cac4cd8bc3e990221c4f28e: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 04:12:05.387019 16890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b8cbc143cac4cd8bc3e990221c4f28e: Bootstrap complete.
I20260924 04:12:05.391057 16890 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b8cbc143cac4cd8bc3e990221c4f28e [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b8cbc143cac4cd8bc3e990221c4f28e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37025 } }
I20260924 04:12:05.391435 16890 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b8cbc143cac4cd8bc3e990221c4f28e [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b8cbc143cac4cd8bc3e990221c4f28e, State: Initialized, Role: FOLLOWER
I20260924 04:12:05.391912 16890 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b8cbc143cac4cd8bc3e990221c4f28e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b8cbc143cac4cd8bc3e990221c4f28e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37025 } }
I20260924 04:12:05.392024 16890 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b8cbc143cac4cd8bc3e990221c4f28e [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:12:05.392105 16890 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b8cbc143cac4cd8bc3e990221c4f28e [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:12:05.392215 16890 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b8cbc143cac4cd8bc3e990221c4f28e [term 2 FOLLOWER]: Advancing to term 3
I20260924 04:12:05.394060 16890 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b8cbc143cac4cd8bc3e990221c4f28e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b8cbc143cac4cd8bc3e990221c4f28e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37025 } }
I20260924 04:12:05.394382 16890 leader_election.cc:302] T 00000000000000000000000000000000 P 4b8cbc143cac4cd8bc3e990221c4f28e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b8cbc143cac4cd8bc3e990221c4f28e; no voters:
I20260924 04:12:05.394747 16890 leader_election.cc:288] T 00000000000000000000000000000000 P 4b8cbc143cac4cd8bc3e990221c4f28e [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 04:12:05.394834 16894 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b8cbc143cac4cd8bc3e990221c4f28e [term 3 FOLLOWER]: Leader election won for term 3
I20260924 04:12:05.395027 16894 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b8cbc143cac4cd8bc3e990221c4f28e [term 3 LEADER]: Becoming Leader. State: Replica: 4b8cbc143cac4cd8bc3e990221c4f28e, State: Running, Role: LEADER
I20260924 04:12:05.395309 16894 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b8cbc143cac4cd8bc3e990221c4f28e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b8cbc143cac4cd8bc3e990221c4f28e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37025 } }
I20260924 04:12:05.395933 16890 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b8cbc143cac4cd8bc3e990221c4f28e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:12:05.396215 16894 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b8cbc143cac4cd8bc3e990221c4f28e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b8cbc143cac4cd8bc3e990221c4f28e. Latest consensus state: current_term: 3 leader_uuid: "4b8cbc143cac4cd8bc3e990221c4f28e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b8cbc143cac4cd8bc3e990221c4f28e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37025 } } }
I20260924 04:12:05.396328 16894 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b8cbc143cac4cd8bc3e990221c4f28e [sys.catalog]: This master's current role is: LEADER
I20260924 04:12:05.396590 16895 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b8cbc143cac4cd8bc3e990221c4f28e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "4b8cbc143cac4cd8bc3e990221c4f28e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b8cbc143cac4cd8bc3e990221c4f28e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37025 } } }
I20260924 04:12:05.396735 16895 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b8cbc143cac4cd8bc3e990221c4f28e [sys.catalog]: This master's current role is: LEADER
I20260924 04:12:05.396999 16898 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:12:05.401980 16898 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:12:05.402891 16898 catalog_manager.cc:1259] Loaded cluster ID: 5cd8e9d1e20a4497b23d8cb482326859
I20260924 04:12:05.402953 16898 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:12:05.405452 16898 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:12:05.406190 16898 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4b8cbc143cac4cd8bc3e990221c4f28e: Loaded TSK: 0
I20260924 04:12:05.406270 16898 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4b8cbc143cac4cd8bc3e990221c4f28e: Loaded TSK: 1
F20260924 04:12:05.407011 16898 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fcf536339ad google::LogMessage::Fail() at ??:0
@ 0x7fcf5363688a google::LogMessage::SendToLog() at ??:0
@ 0x7fcf536334de google::LogMessage::Flush() at ??:0
@ 0x7fcf53634a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fcf562f718d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fcf562eff0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fcf5df9c7bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fcf5df7699c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fcf5df77178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7fcf5dfb8df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fcf5dfd782b std::function<>::operator()() at ??:0
@ 0x7fcf5df76b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fcf5df77f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fcf5df74579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fcf5dfb8640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fcf558853d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fcf55885c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fcf558873f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fcf55876512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fcf56eba6db start_thread at ??:0
@ 0x7fcf5213161f clone at ??:0
W20260924 04:12:05.489807 16892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:12:05.490038 16892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:12:05.490082 16892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:12:05.494405 16892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:12:05.494455 16892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:12:05.494515 16892 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:12:05.500249 16892 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:42515
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=41211
--tserver_master_addrs=127.0.98.62:37025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16892
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:12:05.501431 16892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:12:05.502377 16892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:12:05.511861 16940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:12:05.511868 16938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:12:05.511870 16937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:12:05.512198 16892 server_base.cc:1034] running on GCE node
I20260924 04:12:05.512905 16892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:12:05.513648 16892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:12:05.514847 16892 hybrid_clock.cc:648] HybridClock initialized: now 1790223125514811 us; error 48 us; skew 500 ppm
Sep 24 04:12:05 dist-test-slave-00ld krb5kdc[15746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223125, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:12:05.518182 16892 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:12:05.519061 16892 webserver.cc:460] Webserver started at http://127.0.98.1:41211/ using document root <none> and password file <none>
I20260924 04:12:05.519671 16892 fs_manager.cc:359] Metadata directory not provided
I20260924 04:12:05.519766 16892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:12:05.523890 16892 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 04:12:05.526290 16947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:12:05.527547 16892 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:12:05.527679 16892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-0/wal
uuid: "29fe56f4efe74850870cd033fdea81d3"
format_stamp: "Formatted at 2026-09-24 04:11:23 on dist-test-slave-00ld"
server_key: "a8a753b5000d70b1e941547fbea391b0"
server_key_iv: "ff1443f5b70c7947ef0f1f9a3e1a936b"
server_key_version: "encryptionkey@0"
I20260924 04:12:05.528206 16892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:12:05.547955 16892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:12:05.551213 16892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:12:05.551409 16892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:12:05.551992 16892 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:12:05.553133 16954 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 04:12:05.558565 16892 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 04:12:05.558635 16892 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260924 04:12:05.558710 16892 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 04:12:05.560364 16892 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 04:12:05.560426 16892 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260924 04:12:05.560751 16954 tablet_bootstrap.cc:492] T 53d4138f8ea243c8832b53ebcbb148ab P 29fe56f4efe74850870cd033fdea81d3: Bootstrap starting.
I20260924 04:12:05.570741 16892 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42515
I20260924 04:12:05.572032 16892 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:12:05.572904 17061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42515 every 8 connection(s)
I20260924 04:12:05.577284 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 16892
I20260924 04:12:05.577730 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:38987
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:37025
--webserver_port=36121
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:12:05.582496 16954 log.cc:824] T 53d4138f8ea243c8832b53ebcbb148ab P 29fe56f4efe74850870cd033fdea81d3: Log is configured to *not* fsync() on all Append() calls
I20260924 04:12:05.589344 16954 tablet_bootstrap.cc:492] T 53d4138f8ea243c8832b53ebcbb148ab P 29fe56f4efe74850870cd033fdea81d3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 04:12:05.589730 16954 tablet_bootstrap.cc:492] T 53d4138f8ea243c8832b53ebcbb148ab P 29fe56f4efe74850870cd033fdea81d3: Bootstrap complete.
I20260924 04:12:05.590824 16954 ts_tablet_manager.cc:1397] T 53d4138f8ea243c8832b53ebcbb148ab P 29fe56f4efe74850870cd033fdea81d3: Time spent bootstrapping tablet: real 0.030s user 0.016s sys 0.011s
I20260924 04:12:05.593312 16954 raft_consensus.cc:348] T 53d4138f8ea243c8832b53ebcbb148ab P 29fe56f4efe74850870cd033fdea81d3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59864430ab9443eb82dc451f4a293582" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42569 } } peers { permanent_uuid: "b30b39da74e84f2a8af94206e4a2e8e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38987 } } peers { permanent_uuid: "29fe56f4efe74850870cd033fdea81d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42515 } }
I20260924 04:12:05.593722 16954 raft_consensus.cc:729] T 53d4138f8ea243c8832b53ebcbb148ab P 29fe56f4efe74850870cd033fdea81d3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29fe56f4efe74850870cd033fdea81d3, State: Initialized, Role: FOLLOWER
I20260924 04:12:05.594202 16954 consensus_queue.cc:260] T 53d4138f8ea243c8832b53ebcbb148ab P 29fe56f4efe74850870cd033fdea81d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59864430ab9443eb82dc451f4a293582" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42569 } } peers { permanent_uuid: "b30b39da74e84f2a8af94206e4a2e8e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38987 } } peers { permanent_uuid: "29fe56f4efe74850870cd033fdea81d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42515 } }
I20260924 04:12:05.595122 16954 ts_tablet_manager.cc:1428] T 53d4138f8ea243c8832b53ebcbb148ab P 29fe56f4efe74850870cd033fdea81d3: Time spent starting tablet: real 0.004s user 0.005s sys 0.001s
W20260924 04:12:05.678689 17064 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:12:05.575168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.1:56881)
0924 04:12:05.575645 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:05.575697 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 04:12:05.576528 (+ 831us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:12:05.677908 (+101380us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":154,"threads_started":1}
W20260924 04:12:05.679447 17062 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37025 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:37025: Client connection negotiation failed: client connection to 127.0.98.62:37025: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 04:12:05.680265 17064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:05.679942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.1:33465)
0924 04:12:05.680100 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:05.680190 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260924 04:12:05.681031 17064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:05.680815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.1:43645)
0924 04:12:05.680917 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:05.680969 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260924 04:12:05.681213 17062 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 04:12:05.719990 17065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:12:05.720217 17065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:12:05.720300 17065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:12:05.724213 17065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:12:05.724269 17065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:12:05.724366 17065 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:12:05.729480 17065 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:38987
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=36121
--tserver_master_addrs=127.0.98.62:37025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17065
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:12:05.730429 17065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:12:05.731391 17065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:12:05.740588 17072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:12:05.740622 17073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:12:05.741134 17075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:12:05.741271 17065 server_base.cc:1034] running on GCE node
I20260924 04:12:05.741712 17065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:12:05.742339 17065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:12:05.743499 17065 hybrid_clock.cc:648] HybridClock initialized: now 1790223125743464 us; error 45 us; skew 500 ppm
Sep 24 04:12:05 dist-test-slave-00ld krb5kdc[15746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223125, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:12:05.750676 17065 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:12:05.751716 17065 webserver.cc:460] Webserver started at http://127.0.98.2:36121/ using document root <none> and password file <none>
I20260924 04:12:05.752326 17065 fs_manager.cc:359] Metadata directory not provided
I20260924 04:12:05.752421 17065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:12:05.756438 17065 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:12:05.758637 17082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:12:05.759976 17065 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 04:12:05.760110 17065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-1/wal
uuid: "b30b39da74e84f2a8af94206e4a2e8e2"
format_stamp: "Formatted at 2026-09-24 04:11:23 on dist-test-slave-00ld"
server_key: "26eeade390c1ca46f79b67a27c04fa87"
server_key_iv: "edf98eef0f41d7c98248f520dee3a3b8"
server_key_version: "encryptionkey@0"
I20260924 04:12:05.760679 17065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:12:05.796613 17065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:12:05.799935 17065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:12:05.800168 17065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:12:05.800848 17065 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:12:05.801756 17065 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:12:05.801831 17065 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:12:05.801911 17065 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:12:05.801956 17065 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:12:05.811672 17065 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38987
I20260924 04:12:05.811692 17195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38987 every 8 connection(s)
I20260924 04:12:05.812745 17065 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:12:05.816038 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 17065
I20260924 04:12:05.816495 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:42569
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:37025
--webserver_port=39903
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:12:05.816901 17198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:05.814885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.2:51709)
0924 04:12:05.815383 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:05.815495 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":108,"threads_started":1}
W20260924 04:12:05.817512 17196 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37025 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:37025: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
I20260924 04:12:05.818240 17198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:05.817960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.2:47327)
0924 04:12:05.818099 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:05.818160 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260924 04:12:05.818912 17198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:05.818667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.2:46653)
0924 04:12:05.818781 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:05.818839 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260924 04:12:05.819089 17196 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 04:12:05.927079 17199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:12:05.927321 17199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:12:05.927363 17199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:12:05.931424 17199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:12:05.931475 17199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:12:05.931568 17199 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:12:05.936721 17199 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:42569
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=39903
--tserver_master_addrs=127.0.98.62:37025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17199
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:12:05.937809 17199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:12:05.938740 17199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:12:05.948119 17205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:12:05.948318 17207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:12:05.948362 17199 server_base.cc:1034] running on GCE node
W20260924 04:12:05.948119 17204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:12:05.948906 17199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:12:05.949549 17199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:12:05.950728 17199 hybrid_clock.cc:648] HybridClock initialized: now 1790223125950687 us; error 54 us; skew 500 ppm
Sep 24 04:12:05 dist-test-slave-00ld krb5kdc[15746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223125, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:12:05.953389 17199 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:12:05.954211 17199 webserver.cc:460] Webserver started at http://127.0.98.3:39903/ using document root <none> and password file <none>
I20260924 04:12:05.954771 17199 fs_manager.cc:359] Metadata directory not provided
I20260924 04:12:05.954869 17199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:12:05.958669 17199 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260924 04:12:05.960822 17214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:12:05.961977 17199 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:12:05.962121 17199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-2/wal
uuid: "59864430ab9443eb82dc451f4a293582"
format_stamp: "Formatted at 2026-09-24 04:11:23 on dist-test-slave-00ld"
server_key: "1203e1471249287047c88de5eb16b603"
server_key_iv: "67fc4e595c5546b149140b244bc26258"
server_key_version: "encryptionkey@0"
I20260924 04:12:05.962658 17199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:12:05.986161 17199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:12:05.989270 17199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:12:05.989478 17199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:12:05.990082 17199 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:12:05.990922 17199 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:12:05.990990 17199 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:12:05.991063 17199 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:12:05.991104 17199 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:12:05.999837 17199 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42569
I20260924 04:12:05.999862 17327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42569 every 8 connection(s)
I20260924 04:12:06.000869 17199 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790222898807720-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:12:06.003436 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 17199
I20260924 04:12:06.005120 17330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:06.002929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.3:34629)
0924 04:12:06.003387 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:06.003618 (+ 231us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":150,"threads_started":1}
W20260924 04:12:06.005555 17328 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37025 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:37025: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
I20260924 04:12:06.006080 17330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:06.005888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.3:57805)
0924 04:12:06.005983 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:06.006028 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260924 04:12:06.006610 17330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:06.006439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.3:45505)
0924 04:12:06.006525 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:06.006565 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260924 04:12:06.006825 17328 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 04:12:06.682449 17332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:06.681980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.1:42829)
0924 04:12:06.682266 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:06.682332 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":80,"threads_started":1}
I20260924 04:12:06.819953 17333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:06.819590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.2:49421)
0924 04:12:06.819806 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:06.819857 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":66,"threads_started":1}
I20260924 04:12:07.007715 17334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:07.007344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.3:34705)
0924 04:12:07.007568 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:07.007622 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":70,"threads_started":1}
I20260924 04:12:07.146683 17335 raft_consensus.cc:482] T 53d4138f8ea243c8832b53ebcbb148ab P 29fe56f4efe74850870cd033fdea81d3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:12:07.146936 17335 raft_consensus.cc:504] T 53d4138f8ea243c8832b53ebcbb148ab P 29fe56f4efe74850870cd033fdea81d3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59864430ab9443eb82dc451f4a293582" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42569 } } peers { permanent_uuid: "b30b39da74e84f2a8af94206e4a2e8e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38987 } } peers { permanent_uuid: "29fe56f4efe74850870cd033fdea81d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42515 } }
I20260924 04:12:07.148057 17335 leader_election.cc:288] T 53d4138f8ea243c8832b53ebcbb148ab P 29fe56f4efe74850870cd033fdea81d3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59864430ab9443eb82dc451f4a293582 (127.0.98.3:42569), b30b39da74e84f2a8af94206e4a2e8e2 (127.0.98.2:38987)
Sep 24 04:12:07 dist-test-slave-00ld krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223125, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Sep 24 04:12:07 dist-test-slave-00ld krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223125, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260924 04:12:07.161984 17336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:07.148320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38987 (local address 127.0.98.1:58973)
0924 04:12:07.148656 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:07.148671 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:12:07.148807 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:12:07.151148 (+ 2341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:12:07.151150 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:12:07.151518 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:12:07.151824 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:07.151846 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:07.153170 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:07.153175 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:07.154100 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:07.154109 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:07.154798 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:07.154802 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:07.154964 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:07.155650 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:12:07.155666 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:12:07.157682 (+ 2016us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:12:07.159930 (+ 2248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:12:07.159933 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:12:07.159935 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:12:07.160172 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:12:07.160506 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:12:07.160510 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:12:07.160511 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:12:07.160591 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:12:07.161225 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:12:07.161227 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:12:07.161306 (+ 79us) client_negotiation.cc:769] Sending connection context
0924 04:12:07.161433 (+ 127us) client_negotiation.cc:240] Negotiation successful
0924 04:12:07.161751 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"mutex_wait_us":112,"thread_start_us":88,"threads_started":1}
I20260924 04:12:07.161995 17337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:07.148464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37735 (local address 127.0.98.3:42569)
0924 04:12:07.148865 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:07.148890 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:07.148938 (+ 48us) server_negotiation.cc:407] Connection header received
0924 04:12:07.149851 (+ 913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:07.149859 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:07.150323 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:07.150664 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:12:07.152025 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:07.153047 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:07.154219 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:07.154674 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:07.154843 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:07.157486 (+ 2643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:12:07.157493 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:12:07.157509 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:12:07.157543 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:12:07.159270 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:12:07.159847 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:12:07.159851 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:12:07.159855 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:12:07.160015 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:12:07.160363 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:12:07.160365 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:12:07.160366 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:12:07.160880 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:12:07.160982 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:07.161550 (+ 568us) server_negotiation.cc:299] Negotiation successful
0924 04:12:07.161820 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":88,"threads_started":1}
I20260924 04:12:07.162022 17332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:07.148240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42569 (local address 127.0.98.1:37735)
0924 04:12:07.148439 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:07.148456 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 04:12:07.148567 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:12:07.150728 (+ 2161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:12:07.150733 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:12:07.151247 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:12:07.151839 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:07.151852 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:07.153196 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:07.153202 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:07.154102 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:07.154111 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:07.154810 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:07.154814 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:07.154964 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:07.155667 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:12:07.155679 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:12:07.157266 (+ 1587us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:12:07.159383 (+ 2117us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:12:07.159388 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:12:07.159401 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:12:07.159725 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:12:07.160112 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:12:07.160114 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:12:07.160115 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:12:07.160270 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:12:07.160988 (+ 718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:12:07.160994 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:12:07.161230 (+ 236us) client_negotiation.cc:769] Sending connection context
0924 04:12:07.161435 (+ 205us) client_negotiation.cc:240] Negotiation successful
0924 04:12:07.161837 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"mutex_wait_us":210,"spinlock_wait_cycles":10624}
I20260924 04:12:07.162580 17338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:07.148480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58973 (local address 127.0.98.2:38987)
0924 04:12:07.148916 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:07.148933 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:07.148963 (+ 30us) server_negotiation.cc:407] Connection header received
0924 04:12:07.149851 (+ 888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:07.149859 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:07.150323 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:07.150665 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:12:07.152007 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:07.153034 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:07.154221 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:07.154674 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:07.154843 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:07.157842 (+ 2999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:12:07.157846 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:12:07.157876 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:12:07.157913 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:12:07.159830 (+ 1917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:12:07.160265 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:12:07.160270 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:12:07.160275 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:12:07.160419 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:12:07.160715 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:12:07.160722 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:12:07.160724 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:12:07.161121 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:12:07.161247 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:07.161550 (+ 303us) server_negotiation.cc:299] Negotiation successful
0924 04:12:07.161819 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":89,"threads_started":1}
W20260924 04:12:07.164860 16948 leader_election.cc:341] T 53d4138f8ea243c8832b53ebcbb148ab P 29fe56f4efe74850870cd033fdea81d3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 59864430ab9443eb82dc451f4a293582 (127.0.98.3:42569): Not found: Tablet not found: 53d4138f8ea243c8832b53ebcbb148ab
W20260924 04:12:07.165026 16950 leader_election.cc:341] T 53d4138f8ea243c8832b53ebcbb148ab P 29fe56f4efe74850870cd033fdea81d3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b30b39da74e84f2a8af94206e4a2e8e2 (127.0.98.2:38987): Not found: Tablet not found: 53d4138f8ea243c8832b53ebcbb148ab
I20260924 04:12:07.165119 16950 leader_election.cc:302] T 53d4138f8ea243c8832b53ebcbb148ab P 29fe56f4efe74850870cd033fdea81d3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29fe56f4efe74850870cd033fdea81d3; no voters: 59864430ab9443eb82dc451f4a293582, b30b39da74e84f2a8af94206e4a2e8e2
I20260924 04:12:07.165383 17335 raft_consensus.cc:2726] T 53d4138f8ea243c8832b53ebcbb148ab P 29fe56f4efe74850870cd033fdea81d3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 04:12:07.683635 17339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:07.683217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.1:36961)
0924 04:12:07.683477 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:07.683534 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":74,"threads_started":1}
I20260924 04:12:07.821288 17340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:07.820892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.2:53725)
0924 04:12:07.821138 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:07.821196 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":70,"threads_started":1}
I20260924 04:12:08.008819 17341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:08.008389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.3:34101)
0924 04:12:08.008617 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:08.008670 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":72,"threads_started":1}
I20260924 04:12:08.684880 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:08.684390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.1:50673)
0924 04:12:08.684657 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:08.684781 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":71,"threads_started":1}
I20260924 04:12:08.822398 17343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:08.821959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.2:53481)
0924 04:12:08.822257 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:08.822312 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":76,"threads_started":1}
I20260924 04:12:09.009987 17344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:09.009561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.3:33327)
0924 04:12:09.009837 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:09.009896 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":75,"threads_started":1}
I20260924 04:12:09.099854 17345 raft_consensus.cc:482] T 53d4138f8ea243c8832b53ebcbb148ab P 29fe56f4efe74850870cd033fdea81d3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:12:09.100016 17345 raft_consensus.cc:504] T 53d4138f8ea243c8832b53ebcbb148ab P 29fe56f4efe74850870cd033fdea81d3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59864430ab9443eb82dc451f4a293582" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42569 } } peers { permanent_uuid: "b30b39da74e84f2a8af94206e4a2e8e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38987 } } peers { permanent_uuid: "29fe56f4efe74850870cd033fdea81d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42515 } }
I20260924 04:12:09.100451 17345 leader_election.cc:288] T 53d4138f8ea243c8832b53ebcbb148ab P 29fe56f4efe74850870cd033fdea81d3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59864430ab9443eb82dc451f4a293582 (127.0.98.3:42569), b30b39da74e84f2a8af94206e4a2e8e2 (127.0.98.2:38987)
W20260924 04:12:09.101462 16948 leader_election.cc:341] T 53d4138f8ea243c8832b53ebcbb148ab P 29fe56f4efe74850870cd033fdea81d3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 59864430ab9443eb82dc451f4a293582 (127.0.98.3:42569): Not found: Tablet not found: 53d4138f8ea243c8832b53ebcbb148ab
W20260924 04:12:09.101572 16950 leader_election.cc:341] T 53d4138f8ea243c8832b53ebcbb148ab P 29fe56f4efe74850870cd033fdea81d3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b30b39da74e84f2a8af94206e4a2e8e2 (127.0.98.2:38987): Not found: Tablet not found: 53d4138f8ea243c8832b53ebcbb148ab
I20260924 04:12:09.101631 16950 leader_election.cc:302] T 53d4138f8ea243c8832b53ebcbb148ab P 29fe56f4efe74850870cd033fdea81d3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29fe56f4efe74850870cd033fdea81d3; no voters: 59864430ab9443eb82dc451f4a293582, b30b39da74e84f2a8af94206e4a2e8e2
I20260924 04:12:09.101753 17345 raft_consensus.cc:2726] T 53d4138f8ea243c8832b53ebcbb148ab P 29fe56f4efe74850870cd033fdea81d3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 04:12:09.686076 17346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:09.685656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.1:36879)
0924 04:12:09.685927 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:09.685979 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":97,"threads_started":1}
I20260924 04:12:09.823668 17347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:09.823278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.2:37047)
0924 04:12:09.823525 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:09.823585 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":71,"threads_started":1}
I20260924 04:12:10.011070 17348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:10.010694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.3:60953)
0924 04:12:10.010928 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:10.010981 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":73,"threads_started":1}
I20260924 04:12:10.687331 17349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:10.686921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.1:43363)
0924 04:12:10.687183 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:10.687234 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":70,"threads_started":1}
I20260924 04:12:10.824958 17350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:10.824481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.2:36197)
0924 04:12:10.824775 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:10.824839 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":83,"threads_started":1}
I20260924 04:12:11.012161 17351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:11.011805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.3:56201)
0924 04:12:11.012029 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:11.012078 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":72,"threads_started":1}
I20260924 04:12:11.260650 17352 raft_consensus.cc:482] T 53d4138f8ea243c8832b53ebcbb148ab P 29fe56f4efe74850870cd033fdea81d3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:12:11.260843 17352 raft_consensus.cc:504] T 53d4138f8ea243c8832b53ebcbb148ab P 29fe56f4efe74850870cd033fdea81d3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59864430ab9443eb82dc451f4a293582" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42569 } } peers { permanent_uuid: "b30b39da74e84f2a8af94206e4a2e8e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38987 } } peers { permanent_uuid: "29fe56f4efe74850870cd033fdea81d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42515 } }
I20260924 04:12:11.261204 17352 leader_election.cc:288] T 53d4138f8ea243c8832b53ebcbb148ab P 29fe56f4efe74850870cd033fdea81d3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59864430ab9443eb82dc451f4a293582 (127.0.98.3:42569), b30b39da74e84f2a8af94206e4a2e8e2 (127.0.98.2:38987)
W20260924 04:12:11.262027 16948 leader_election.cc:341] T 53d4138f8ea243c8832b53ebcbb148ab P 29fe56f4efe74850870cd033fdea81d3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 59864430ab9443eb82dc451f4a293582 (127.0.98.3:42569): Not found: Tablet not found: 53d4138f8ea243c8832b53ebcbb148ab
W20260924 04:12:11.262216 16950 leader_election.cc:341] T 53d4138f8ea243c8832b53ebcbb148ab P 29fe56f4efe74850870cd033fdea81d3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b30b39da74e84f2a8af94206e4a2e8e2 (127.0.98.2:38987): Not found: Tablet not found: 53d4138f8ea243c8832b53ebcbb148ab
I20260924 04:12:11.262282 16950 leader_election.cc:302] T 53d4138f8ea243c8832b53ebcbb148ab P 29fe56f4efe74850870cd033fdea81d3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29fe56f4efe74850870cd033fdea81d3; no voters: 59864430ab9443eb82dc451f4a293582, b30b39da74e84f2a8af94206e4a2e8e2
I20260924 04:12:11.262414 17352 raft_consensus.cc:2726] T 53d4138f8ea243c8832b53ebcbb148ab P 29fe56f4efe74850870cd033fdea81d3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 04:12:11.688460 17353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:11.688031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.1:52111)
0924 04:12:11.688305 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:11.688363 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":78,"threads_started":1}
I20260924 04:12:11.826341 17354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:11.825766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.2:50481)
0924 04:12:11.826098 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:11.826238 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":107,"threads_started":1}
I20260924 04:12:12.013406 17355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:12.013019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.3:39043)
0924 04:12:12.013250 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:12.013318 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":71,"threads_started":1}
I20260924 04:12:12.689744 17356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:12.689330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.1:38253)
0924 04:12:12.689589 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:12.689647 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":74,"threads_started":1}
I20260924 04:12:12.827551 17357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:12.827080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.2:34675)
0924 04:12:12.827378 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:12.827443 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":78,"threads_started":1}
I20260924 04:12:13.014622 17358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:13.014173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.3:56613)
0924 04:12:13.014469 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:13.014527 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":67,"threads_started":1}
I20260924 04:12:13.690872 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:13.690475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.1:49777)
0924 04:12:13.690722 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:13.690776 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":91,"threads_started":1}
I20260924 04:12:13.828940 17360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:13.828424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.2:46235)
0924 04:12:13.828758 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:13.828843 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":88,"threads_started":1}
I20260924 04:12:14.016292 17361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:14.015476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.3:42477)
0924 04:12:14.015892 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:14.016051 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":85,"threads_started":1}
I20260924 04:12:14.692041 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:14.691625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.1:50153)
0924 04:12:14.691882 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:14.691940 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":73,"threads_started":1}
I20260924 04:12:14.830161 17363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:14.829739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.2:45699)
0924 04:12:14.830004 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:14.830067 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":75,"threads_started":1}
I20260924 04:12:15.017975 17364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:15.017441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.3:37723)
0924 04:12:15.017723 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:15.017846 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":75,"threads_started":1}
I20260924 04:12:15.693596 17365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:15.693162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.1:38847)
0924 04:12:15.693435 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:15.693499 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":78,"threads_started":1}
I20260924 04:12:15.804117 17366 raft_consensus.cc:482] T 53d4138f8ea243c8832b53ebcbb148ab P 29fe56f4efe74850870cd033fdea81d3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:12:15.804251 17366 raft_consensus.cc:504] T 53d4138f8ea243c8832b53ebcbb148ab P 29fe56f4efe74850870cd033fdea81d3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59864430ab9443eb82dc451f4a293582" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42569 } } peers { permanent_uuid: "b30b39da74e84f2a8af94206e4a2e8e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38987 } } peers { permanent_uuid: "29fe56f4efe74850870cd033fdea81d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42515 } }
I20260924 04:12:15.804757 17366 leader_election.cc:288] T 53d4138f8ea243c8832b53ebcbb148ab P 29fe56f4efe74850870cd033fdea81d3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59864430ab9443eb82dc451f4a293582 (127.0.98.3:42569), b30b39da74e84f2a8af94206e4a2e8e2 (127.0.98.2:38987)
W20260924 04:12:15.805508 16948 leader_election.cc:341] T 53d4138f8ea243c8832b53ebcbb148ab P 29fe56f4efe74850870cd033fdea81d3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 59864430ab9443eb82dc451f4a293582 (127.0.98.3:42569): Not found: Tablet not found: 53d4138f8ea243c8832b53ebcbb148ab
W20260924 04:12:15.805610 16950 leader_election.cc:341] T 53d4138f8ea243c8832b53ebcbb148ab P 29fe56f4efe74850870cd033fdea81d3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b30b39da74e84f2a8af94206e4a2e8e2 (127.0.98.2:38987): Not found: Tablet not found: 53d4138f8ea243c8832b53ebcbb148ab
I20260924 04:12:15.805728 16950 leader_election.cc:302] T 53d4138f8ea243c8832b53ebcbb148ab P 29fe56f4efe74850870cd033fdea81d3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29fe56f4efe74850870cd033fdea81d3; no voters: 59864430ab9443eb82dc451f4a293582, b30b39da74e84f2a8af94206e4a2e8e2
I20260924 04:12:15.805891 17366 raft_consensus.cc:2726] T 53d4138f8ea243c8832b53ebcbb148ab P 29fe56f4efe74850870cd033fdea81d3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 04:12:15.831565 17367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:15.831144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.2:39297)
0924 04:12:15.831415 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:15.831477 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":98,"threads_started":1}
I20260924 04:12:16.019377 17368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:16.018940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.3:42107)
0924 04:12:16.019215 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:16.019278 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":72,"threads_started":1}
I20260924 04:12:16.694954 17369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:16.694494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.1:46255)
0924 04:12:16.694779 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:16.694856 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":80,"threads_started":1}
I20260924 04:12:16.832864 17370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:16.832386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.2:41721)
0924 04:12:16.832643 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:16.832759 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":75,"threads_started":1}
I20260924 04:12:17.020790 17371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:17.020271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.3:45843)
0924 04:12:17.020540 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:17.020604 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":76,"threads_started":1}
I20260924 04:12:17.696120 17372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:17.695689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.1:43325)
0924 04:12:17.695959 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:17.696012 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":88,"threads_started":1}
I20260924 04:12:17.834052 17373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:17.833631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.2:38151)
0924 04:12:17.833879 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:17.833938 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":76,"threads_started":1}
I20260924 04:12:18.022017 17374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:18.021562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.3:51137)
0924 04:12:18.021866 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:18.021924 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":138,"threads_started":1}
I20260924 04:12:18.697642 17375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:18.697172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.1:47199)
0924 04:12:18.697483 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:18.697539 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":136,"threads_started":1}
I20260924 04:12:18.835333 17376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:18.834925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.2:60637)
0924 04:12:18.835184 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:18.835240 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":84,"threads_started":1}
I20260924 04:12:19.023473 17377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:19.022966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.3:55899)
0924 04:12:19.023291 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:19.023355 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":137,"threads_started":1}
I20260924 04:12:19.698974 17378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:19.698486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.1:48975)
0924 04:12:19.698805 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:19.698869 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":91,"threads_started":1}
I20260924 04:12:19.836752 17379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:19.836229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.2:42739)
0924 04:12:19.836525 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:19.836584 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":77,"threads_started":1}
I20260924 04:12:20.024991 17380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:20.024409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.3:51995)
0924 04:12:20.024670 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:20.024895 (+ 225us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":82,"threads_started":1}
I20260924 04:12:20.700428 17381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:20.699999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.1:56485)
0924 04:12:20.700258 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:20.700331 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":88,"threads_started":1}
I20260924 04:12:20.838272 17382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:20.837588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37025 (local address 127.0.98.2:56599)
0924 04:12:20.837832 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:20.837888 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":68,"threads_started":1}
I20260924 04:12:21.011345 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 16892
I20260924 04:12:21.017411 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 17065
I20260924 04:12:21.022805 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 17199
2026-09-24T04:12:21Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59203 ms)
[----------] 30 tests from SecurityITest (242208 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222898807720-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:12:21 dist-test-slave-00ld krb5kdc[17385](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:12:21 dist-test-slave-00ld krb5kdc[17385](info): set up 2 sockets
Sep 24 04:12:21 dist-test-slave-00ld krb5kdc[17385](info): commencing operation
krb5kdc: starting...
W20260924 04:12:22.072556 392 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222898807720-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222898807720-392-0/krb5kdc/test-user.keytab.
Sep 24 04:12:22 dist-test-slave-00ld krb5kdc[17385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223142, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:12:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:12:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:12:22.193434 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222898807720-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222898807720-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222898807720-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33709
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33709
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:12:22.302354 17398 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:12:22.302629 17398 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:12:22.302703 17398 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:12:22.306613 17398 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:12:22.306697 17398 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:12:22.306734 17398 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:12:22.306792 17398 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:12:22.306844 17398 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:12:22.312042 17398 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222898807720-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222898807720-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33709
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17398
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222898807720-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:12:22.313110 17398 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:12:22.314142 17398 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:12:22.324816 17403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:12:22.324769 17404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:12:22.324944 17406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:12:22.325024 17398 server_base.cc:1034] running on GCE node
I20260924 04:12:22.325675 17398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:12:22.326810 17398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:12:22.328012 17398 hybrid_clock.cc:648] HybridClock initialized: now 1790223142327984 us; error 40 us; skew 500 ppm
Sep 24 04:12:22 dist-test-slave-00ld krb5kdc[17385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223142, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:12:22.330940 17398 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:12:22.332217 17398 webserver.cc:460] Webserver started at http://127.0.98.62:41303/ using document root <none> and password file <none>
I20260924 04:12:22.332858 17398 fs_manager.cc:359] Metadata directory not provided
I20260924 04:12:22.332953 17398 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:12:22.333173 17398 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:12:22.335182 17398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222898807720-392-0/minicluster-data/master-0/data/instance:
uuid: "a07ac4d42fc54e93ab69ff8b39bcd60d"
format_stamp: "Formatted at 2026-09-24 04:12:22 on dist-test-slave-00ld"
server_key: "0948ebd3b1d0362718d79221d18f9d4d"
server_key_iv: "6d44e30437d26f58114868854e0384ed"
server_key_version: "encryptionkey@0"
I20260924 04:12:22.335747 17398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222898807720-392-0/minicluster-data/master-0/wal/instance:
uuid: "a07ac4d42fc54e93ab69ff8b39bcd60d"
format_stamp: "Formatted at 2026-09-24 04:12:22 on dist-test-slave-00ld"
server_key: "0948ebd3b1d0362718d79221d18f9d4d"
server_key_iv: "6d44e30437d26f58114868854e0384ed"
server_key_version: "encryptionkey@0"
I20260924 04:12:22.339486 17398 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260924 04:12:22.342087 17413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:12:22.343375 17398 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:12:22.343564 17398 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222898807720-392-0/minicluster-data/master-0/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222898807720-392-0/minicluster-data/master-0/wal
uuid: "a07ac4d42fc54e93ab69ff8b39bcd60d"
format_stamp: "Formatted at 2026-09-24 04:12:22 on dist-test-slave-00ld"
server_key: "0948ebd3b1d0362718d79221d18f9d4d"
server_key_iv: "6d44e30437d26f58114868854e0384ed"
server_key_version: "encryptionkey@0"
I20260924 04:12:22.343694 17398 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222898807720-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222898807720-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222898807720-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:12:22.375222 17398 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:12:22.378659 17398 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:12:22.378876 17398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:12:22.386080 17398 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33709
I20260924 04:12:22.386101 17465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33709 every 8 connection(s)
I20260924 04:12:22.387001 17398 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:12:22.389492 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 17398
I20260924 04:12:22.389639 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222898807720-392-0/minicluster-data/master-0/wal/instance
I20260924 04:12:22.389802 17466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:12:22.389973 392 external_mini_cluster.cc:1428] Setting key 2362c1f99bfa1c0d32fdb80bfba5b767
I20260924 04:12:22.396248 17466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a07ac4d42fc54e93ab69ff8b39bcd60d: Bootstrap starting.
I20260924 04:12:22.398501 17466 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a07ac4d42fc54e93ab69ff8b39bcd60d: Neither blocks nor log segments found. Creating new log.
Sep 24 04:12:22 dist-test-slave-00ld krb5kdc[17385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223142, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:12:22.399303 17466 log.cc:824] T 00000000000000000000000000000000 P a07ac4d42fc54e93ab69ff8b39bcd60d: Log is configured to *not* fsync() on all Append() calls
I20260924 04:12:22.401732 17466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a07ac4d42fc54e93ab69ff8b39bcd60d: No bootstrap required, opened a new log
I20260924 04:12:22.404237 17469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:22.391421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48768 (local address 127.0.98.62:33709)
0924 04:12:22.391846 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:22.391861 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:22.391894 (+ 33us) server_negotiation.cc:407] Connection header received
0924 04:12:22.392519 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:22.392535 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:22.392929 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:22.393250 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:12:22.394060 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:22.395134 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:22.395971 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:22.396516 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:22.396679 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:22.399134 (+ 2455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:12:22.399140 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:12:22.399157 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:12:22.399200 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:12:22.401182 (+ 1982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:12:22.401584 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:12:22.401589 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:12:22.401594 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:12:22.401725 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:12:22.402056 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:12:22.402059 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:12:22.402060 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:12:22.402549 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:12:22.402743 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:22.402935 (+ 192us) server_negotiation.cc:299] Negotiation successful
0924 04:12:22.403235 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":152,"threads_started":1}
I20260924 04:12:22.406100 17466 raft_consensus.cc:348] T 00000000000000000000000000000000 P a07ac4d42fc54e93ab69ff8b39bcd60d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a07ac4d42fc54e93ab69ff8b39bcd60d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33709 } }
I20260924 04:12:22.406390 17466 raft_consensus.cc:374] T 00000000000000000000000000000000 P a07ac4d42fc54e93ab69ff8b39bcd60d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:12:22.406502 17466 raft_consensus.cc:729] T 00000000000000000000000000000000 P a07ac4d42fc54e93ab69ff8b39bcd60d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a07ac4d42fc54e93ab69ff8b39bcd60d, State: Initialized, Role: FOLLOWER
I20260924 04:12:22.406986 17466 consensus_queue.cc:260] T 00000000000000000000000000000000 P a07ac4d42fc54e93ab69ff8b39bcd60d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a07ac4d42fc54e93ab69ff8b39bcd60d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33709 } }
I20260924 04:12:22.407140 17466 raft_consensus.cc:388] T 00000000000000000000000000000000 P a07ac4d42fc54e93ab69ff8b39bcd60d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:12:22.407207 17466 raft_consensus.cc:482] T 00000000000000000000000000000000 P a07ac4d42fc54e93ab69ff8b39bcd60d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:12:22.407303 17466 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a07ac4d42fc54e93ab69ff8b39bcd60d [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:12:22.408206 17466 raft_consensus.cc:504] T 00000000000000000000000000000000 P a07ac4d42fc54e93ab69ff8b39bcd60d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a07ac4d42fc54e93ab69ff8b39bcd60d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33709 } }
I20260924 04:12:22.408542 17466 leader_election.cc:302] T 00000000000000000000000000000000 P a07ac4d42fc54e93ab69ff8b39bcd60d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a07ac4d42fc54e93ab69ff8b39bcd60d; no voters:
I20260924 04:12:22.408864 17466 leader_election.cc:288] T 00000000000000000000000000000000 P a07ac4d42fc54e93ab69ff8b39bcd60d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:12:22.408975 17471 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a07ac4d42fc54e93ab69ff8b39bcd60d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:12:22.409196 17471 raft_consensus.cc:686] T 00000000000000000000000000000000 P a07ac4d42fc54e93ab69ff8b39bcd60d [term 1 LEADER]: Becoming Leader. State: Replica: a07ac4d42fc54e93ab69ff8b39bcd60d, State: Running, Role: LEADER
I20260924 04:12:22.409571 17471 consensus_queue.cc:237] T 00000000000000000000000000000000 P a07ac4d42fc54e93ab69ff8b39bcd60d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a07ac4d42fc54e93ab69ff8b39bcd60d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33709 } }
I20260924 04:12:22.410188 17466 sys_catalog.cc:564] T 00000000000000000000000000000000 P a07ac4d42fc54e93ab69ff8b39bcd60d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:12:22.411204 17472 sys_catalog.cc:455] T 00000000000000000000000000000000 P a07ac4d42fc54e93ab69ff8b39bcd60d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a07ac4d42fc54e93ab69ff8b39bcd60d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a07ac4d42fc54e93ab69ff8b39bcd60d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33709 } } }
I20260924 04:12:22.411346 17472 sys_catalog.cc:458] T 00000000000000000000000000000000 P a07ac4d42fc54e93ab69ff8b39bcd60d [sys.catalog]: This master's current role is: LEADER
I20260924 04:12:22.411741 17480 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:12:22.411741 17473 sys_catalog.cc:455] T 00000000000000000000000000000000 P a07ac4d42fc54e93ab69ff8b39bcd60d [sys.catalog]: SysCatalogTable state changed. Reason: New leader a07ac4d42fc54e93ab69ff8b39bcd60d. Latest consensus state: current_term: 1 leader_uuid: "a07ac4d42fc54e93ab69ff8b39bcd60d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a07ac4d42fc54e93ab69ff8b39bcd60d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33709 } } }
I20260924 04:12:22.412398 17473 sys_catalog.cc:458] T 00000000000000000000000000000000 P a07ac4d42fc54e93ab69ff8b39bcd60d [sys.catalog]: This master's current role is: LEADER
I20260924 04:12:22.414525 17480 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:12:22.420182 17480 catalog_manager.cc:1347] Generated new cluster ID: 37e6ab70702848269d407e6fb5d75f38
I20260924 04:12:22.420252 17480 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:12:22.464833 17480 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:12:22.466178 17480 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:12:22.471256 17480 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a07ac4d42fc54e93ab69ff8b39bcd60d: Generated new TSK 0
I20260924 04:12:22.471976 17480 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222898807720-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222898807720-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:12:22.514148 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222898807720-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222898807720-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222898807720-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222898807720-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33709
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222898807720-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:12:22.623320 17492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:12:22.623610 17492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:12:22.623682 17492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:12:22.627674 17492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:12:22.627767 17492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:12:22.627858 17492 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:12:22.633085 17492 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222898807720-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222898807720-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222898807720-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222898807720-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17492
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222898807720-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:12:22.634104 17492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:12:22.635414 17492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:12:22.645604 17500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:12:22.645604 17497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:12:22.645630 17498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:12:22.645840 17492 server_base.cc:1034] running on GCE node
I20260924 04:12:22.646375 17492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:12:22.647058 17492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:12:22.648247 17492 hybrid_clock.cc:648] HybridClock initialized: now 1790223142648220 us; error 46 us; skew 500 ppm
Sep 24 04:12:22 dist-test-slave-00ld krb5kdc[17385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223142, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:12:22.650988 17492 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:12:22.651847 17492 webserver.cc:460] Webserver started at http://127.0.98.1:38553/ using document root <none> and password file <none>
I20260924 04:12:22.652405 17492 fs_manager.cc:359] Metadata directory not provided
I20260924 04:12:22.652496 17492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:12:22.652782 17492 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:12:22.654748 17492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222898807720-392-0/minicluster-data/ts-0/data/instance:
uuid: "ff5f2ef59b7d46488f6ce346e80120ac"
format_stamp: "Formatted at 2026-09-24 04:12:22 on dist-test-slave-00ld"
server_key: "77bf8d45d58df19c0c9db862b3a3c385"
server_key_iv: "257719a35283cb080ecbebd81caff551"
server_key_version: "encryptionkey@0"
I20260924 04:12:22.655295 17492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222898807720-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ff5f2ef59b7d46488f6ce346e80120ac"
format_stamp: "Formatted at 2026-09-24 04:12:22 on dist-test-slave-00ld"
server_key: "77bf8d45d58df19c0c9db862b3a3c385"
server_key_iv: "257719a35283cb080ecbebd81caff551"
server_key_version: "encryptionkey@0"
I20260924 04:12:22.658983 17492 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 04:12:22.661372 17507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:12:22.662559 17492 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:12:22.662693 17492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222898807720-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222898807720-392-0/minicluster-data/ts-0/wal
uuid: "ff5f2ef59b7d46488f6ce346e80120ac"
format_stamp: "Formatted at 2026-09-24 04:12:22 on dist-test-slave-00ld"
server_key: "77bf8d45d58df19c0c9db862b3a3c385"
server_key_iv: "257719a35283cb080ecbebd81caff551"
server_key_version: "encryptionkey@0"
I20260924 04:12:22.662801 17492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222898807720-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222898807720-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222898807720-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:12:22.681660 17492 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:12:22.684942 17492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:12:22.685143 17492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:12:22.685719 17492 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:12:22.686506 17492 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:12:22.686576 17492 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:12:22.686650 17492 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:12:22.686703 17492 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:12:22.695284 17492 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39695
I20260924 04:12:22.695312 17620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39695 every 8 connection(s)
I20260924 04:12:22.696256 17492 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222898807720-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:12:22.700650 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 17492
I20260924 04:12:22.700878 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222898807720-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:12:22.701171 392 external_mini_cluster.cc:1428] Setting key 5d95a76fffa7dbb626b792489989e9af
Sep 24 04:12:22 dist-test-slave-00ld krb5kdc[17385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223142, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 04:12:22.710207 17469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:22.698115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60065 (local address 127.0.98.62:33709)
0924 04:12:22.698254 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:22.698267 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:22.699124 (+ 857us) server_negotiation.cc:407] Connection header received
0924 04:12:22.700432 (+ 1308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:22.700436 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:22.700492 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:22.700574 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:12:22.701907 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:22.702451 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:22.703601 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:22.703841 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:22.703950 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:22.706632 (+ 2682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:12:22.706635 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:12:22.706637 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:12:22.706659 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:12:22.708083 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:12:22.708581 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:12:22.708586 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:12:22.708588 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:12:22.708667 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:12:22.709054 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:12:22.709058 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:12:22.709060 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:12:22.709239 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:12:22.709352 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:22.709924 (+ 572us) server_negotiation.cc:299] Negotiation successful
0924 04:12:22.710049 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 04:12:22.711144 17623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:22.698541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.1:60065)
0924 04:12:22.698972 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:22.699014 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 04:12:22.700217 (+ 1203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:12:22.700803 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:12:22.700810 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:12:22.701225 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:12:22.701754 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:22.701775 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:22.702579 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:22.702583 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:22.703471 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:22.703481 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:22.703990 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:22.703993 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:22.704163 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:22.704817 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:12:22.704833 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:12:22.706460 (+ 1627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:12:22.708210 (+ 1750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:12:22.708216 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:12:22.708231 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:12:22.708481 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:12:22.708811 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:12:22.708816 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:12:22.708818 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:12:22.708960 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:12:22.709351 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:12:22.709358 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:12:22.709603 (+ 245us) client_negotiation.cc:769] Sending connection context
0924 04:12:22.709882 (+ 279us) client_negotiation.cc:240] Negotiation successful
0924 04:12:22.710212 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":116,"threads_started":1}
I20260924 04:12:22.712426 17621 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33709
I20260924 04:12:22.712666 17621 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 04:12:22.713218 17621 heartbeater.cc:511] Master 127.0.98.62:33709 requested a full tablet report, sending...
I20260924 04:12:22.714664 17430 ts_manager.cc:194] Registered new tserver with Master: ff5f2ef59b7d46488f6ce346e80120ac (127.0.98.1:39695)
I20260924 04:12:22.715755 17430 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60065
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222898807720-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222898807720-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:12:22.727662 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222898807720-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222898807720-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222898807720-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222898807720-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33709
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222898807720-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:12:22.835708 17626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:12:22.835971 17626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:12:22.836015 17626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:12:22.840004 17626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:12:22.840071 17626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:12:22.840137 17626 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:12:22.845549 17626 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222898807720-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222898807720-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222898807720-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222898807720-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17626
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222898807720-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:12:22.846544 17626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:12:22.847556 17626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:12:22.857008 17626 server_base.cc:1034] running on GCE node
W20260924 04:12:22.857035 17631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:12:22.857026 17632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:12:22.857252 17634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:12:22.857709 17626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:12:22.858342 17626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:12:22.859501 17626 hybrid_clock.cc:648] HybridClock initialized: now 1790223142859482 us; error 31 us; skew 500 ppm
Sep 24 04:12:22 dist-test-slave-00ld krb5kdc[17385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223142, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:12:22.862176 17626 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:12:22.863065 17626 webserver.cc:460] Webserver started at http://127.0.98.2:37309/ using document root <none> and password file <none>
I20260924 04:12:22.863636 17626 fs_manager.cc:359] Metadata directory not provided
I20260924 04:12:22.863726 17626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:12:22.863961 17626 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:12:22.866120 17626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222898807720-392-0/minicluster-data/ts-1/data/instance:
uuid: "13b187a103584682ab31e9a1119dbf45"
format_stamp: "Formatted at 2026-09-24 04:12:22 on dist-test-slave-00ld"
server_key: "7ce5451c362a14c092dcab9d68a4df49"
server_key_iv: "87823a329b6ac8a9815b75e6271a2c64"
server_key_version: "encryptionkey@0"
I20260924 04:12:22.866744 17626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222898807720-392-0/minicluster-data/ts-1/wal/instance:
uuid: "13b187a103584682ab31e9a1119dbf45"
format_stamp: "Formatted at 2026-09-24 04:12:22 on dist-test-slave-00ld"
server_key: "7ce5451c362a14c092dcab9d68a4df49"
server_key_iv: "87823a329b6ac8a9815b75e6271a2c64"
server_key_version: "encryptionkey@0"
I20260924 04:12:22.870651 17626 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 04:12:22.873127 17641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:12:22.874398 17626 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:12:22.874541 17626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222898807720-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222898807720-392-0/minicluster-data/ts-1/wal
uuid: "13b187a103584682ab31e9a1119dbf45"
format_stamp: "Formatted at 2026-09-24 04:12:22 on dist-test-slave-00ld"
server_key: "7ce5451c362a14c092dcab9d68a4df49"
server_key_iv: "87823a329b6ac8a9815b75e6271a2c64"
server_key_version: "encryptionkey@0"
I20260924 04:12:22.874652 17626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222898807720-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222898807720-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222898807720-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:12:22.917115 17626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:12:22.920053 17626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:12:22.920285 17626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:12:22.920944 17626 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:12:22.921788 17626 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:12:22.921857 17626 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:12:22.921927 17626 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:12:22.921979 17626 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:12:22.930863 17626 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40215
I20260924 04:12:22.930896 17754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40215 every 8 connection(s)
I20260924 04:12:22.931792 17626 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222898807720-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:12:22.934163 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 17626
I20260924 04:12:22.934305 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222898807720-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:12:22.934638 392 external_mini_cluster.cc:1428] Setting key 56cf6f361c003eeab8f681b7428ef563
Sep 24 04:12:22 dist-test-slave-00ld krb5kdc[17385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223142, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260924 04:12:22.948072 17469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:22.933719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50435 (local address 127.0.98.62:33709)
0924 04:12:22.933845 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:22.933851 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:22.935454 (+ 1603us) server_negotiation.cc:407] Connection header received
0924 04:12:22.937158 (+ 1704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:22.937160 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:22.937204 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:22.937277 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:12:22.938690 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:22.939560 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:22.940846 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:22.941295 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:22.941427 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:22.944252 (+ 2825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:12:22.944256 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:12:22.944259 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:12:22.944286 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:12:22.945762 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:12:22.946241 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:12:22.946243 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:12:22.946244 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:12:22.946320 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:12:22.946700 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:12:22.946703 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:12:22.946705 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:12:22.946967 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:12:22.947056 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:22.947766 (+ 710us) server_negotiation.cc:299] Negotiation successful
0924 04:12:22.947894 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 04:12:22.949100 17757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:22.934101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.2:50435)
0924 04:12:22.935262 (+ 1161us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:22.935315 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 04:12:22.936919 (+ 1604us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:12:22.937473 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:12:22.937485 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:12:22.938027 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:12:22.938544 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:22.938559 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:22.939696 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:22.939699 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:22.940725 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:22.940736 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:22.941438 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:22.941441 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:22.941578 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:22.942301 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:12:22.942318 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:12:22.944059 (+ 1741us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:12:22.945877 (+ 1818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:12:22.945882 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:12:22.945893 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:12:22.946143 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:12:22.946407 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:12:22.946409 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:12:22.946411 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:12:22.946539 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:12:22.947117 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:12:22.947123 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:12:22.947491 (+ 368us) client_negotiation.cc:769] Sending connection context
0924 04:12:22.947718 (+ 227us) client_negotiation.cc:240] Negotiation successful
0924 04:12:22.948052 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":926,"thread_start_us":147,"threads_started":1}
I20260924 04:12:22.950361 17755 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33709
I20260924 04:12:22.950615 17755 heartbeater.cc:464] Registering TS with master...
I20260924 04:12:22.951200 17755 heartbeater.cc:511] Master 127.0.98.62:33709 requested a full tablet report, sending...
I20260924 04:12:22.952540 17430 ts_manager.cc:194] Registered new tserver with Master: 13b187a103584682ab31e9a1119dbf45 (127.0.98.2:40215)
I20260924 04:12:22.953300 17430 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50435
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222898807720-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222898807720-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:12:22.970127 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222898807720-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222898807720-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222898807720-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222898807720-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33709
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222898807720-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:12:23.078930 17760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:12:23.079207 17760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:12:23.079280 17760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:12:23.083280 17760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:12:23.083362 17760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:12:23.083455 17760 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:12:23.088835 17760 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222898807720-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222898807720-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222898807720-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222898807720-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17760
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222898807720-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:12:23.089841 17760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:12:23.090870 17760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:12:23.100137 17765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:12:23.100131 17766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:12:23.100127 17768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:12:23.100358 17760 server_base.cc:1034] running on GCE node
I20260924 04:12:23.100812 17760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:12:23.101415 17760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:12:23.102597 17760 hybrid_clock.cc:648] HybridClock initialized: now 1790223143102588 us; error 37 us; skew 500 ppm
Sep 24 04:12:23 dist-test-slave-00ld krb5kdc[17385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223143, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:12:23.105225 17760 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:12:23.106027 17760 webserver.cc:460] Webserver started at http://127.0.98.3:34803/ using document root <none> and password file <none>
I20260924 04:12:23.106556 17760 fs_manager.cc:359] Metadata directory not provided
I20260924 04:12:23.106643 17760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:12:23.106882 17760 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:12:23.108886 17760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222898807720-392-0/minicluster-data/ts-2/data/instance:
uuid: "5691717cbc9f4ee5ad601d6dcf1cb931"
format_stamp: "Formatted at 2026-09-24 04:12:23 on dist-test-slave-00ld"
server_key: "88f7a82cf5bec078af2fe4c5f5f4d265"
server_key_iv: "329dfa2639523483e4d1e1b070c6b506"
server_key_version: "encryptionkey@0"
I20260924 04:12:23.109438 17760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222898807720-392-0/minicluster-data/ts-2/wal/instance:
uuid: "5691717cbc9f4ee5ad601d6dcf1cb931"
format_stamp: "Formatted at 2026-09-24 04:12:23 on dist-test-slave-00ld"
server_key: "88f7a82cf5bec078af2fe4c5f5f4d265"
server_key_iv: "329dfa2639523483e4d1e1b070c6b506"
server_key_version: "encryptionkey@0"
I20260924 04:12:23.113297 17760 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 04:12:23.115640 17775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:12:23.116991 17760 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:12:23.117113 17760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222898807720-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222898807720-392-0/minicluster-data/ts-2/wal
uuid: "5691717cbc9f4ee5ad601d6dcf1cb931"
format_stamp: "Formatted at 2026-09-24 04:12:23 on dist-test-slave-00ld"
server_key: "88f7a82cf5bec078af2fe4c5f5f4d265"
server_key_iv: "329dfa2639523483e4d1e1b070c6b506"
server_key_version: "encryptionkey@0"
I20260924 04:12:23.117213 17760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222898807720-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222898807720-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222898807720-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:12:23.142418 17760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:12:23.145354 17760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:12:23.145550 17760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:12:23.146149 17760 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:12:23.147006 17760 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:12:23.147053 17760 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:12:23.147094 17760 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:12:23.147145 17760 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:12:23.156141 17760 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42745
I20260924 04:12:23.156220 17888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42745 every 8 connection(s)
I20260924 04:12:23.157078 17760 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222898807720-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:12:23.166254 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 17760
I20260924 04:12:23.166412 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222898807720-392-0/minicluster-data/ts-2/wal/instance
Sep 24 04:12:23 dist-test-slave-00ld krb5kdc[17385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223143, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:12:23.166738 392 external_mini_cluster.cc:1428] Setting key a2dd8206df94ea528505ceefdfdef84f
I20260924 04:12:23.170866 17469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:23.158904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52439 (local address 127.0.98.62:33709)
0924 04:12:23.159050 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:23.159054 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:23.159883 (+ 829us) server_negotiation.cc:407] Connection header received
0924 04:12:23.161321 (+ 1438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:23.161324 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:23.161382 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:23.161487 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:12:23.162618 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:23.163151 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:23.164288 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:23.164511 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:23.164623 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:23.167432 (+ 2809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:12:23.167436 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:12:23.167441 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:12:23.167472 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:12:23.169087 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:12:23.169628 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:12:23.169630 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:12:23.169632 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:12:23.169712 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:12:23.169982 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:12:23.169984 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:12:23.169985 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:12:23.170135 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:12:23.170205 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:23.170650 (+ 445us) server_negotiation.cc:299] Negotiation successful
0924 04:12:23.170753 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 04:12:23.171659 17891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:23.159295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33709 (local address 127.0.98.3:52439)
0924 04:12:23.159730 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:23.159771 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 04:12:23.161111 (+ 1340us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:12:23.161656 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:12:23.161663 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:12:23.162015 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:12:23.162475 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:23.162487 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:23.163276 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:23.163282 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:23.164162 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:23.164171 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:23.164659 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:23.164663 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:23.164859 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:23.165467 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:12:23.165484 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:12:23.167271 (+ 1787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:12:23.169212 (+ 1941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:12:23.169216 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:12:23.169228 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:12:23.169527 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:12:23.169790 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:12:23.169792 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:12:23.169793 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:12:23.169917 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:12:23.170232 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:12:23.170237 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:12:23.170455 (+ 218us) client_negotiation.cc:769] Sending connection context
0924 04:12:23.170603 (+ 148us) client_negotiation.cc:240] Negotiation successful
0924 04:12:23.170864 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":114,"threads_started":1}
I20260924 04:12:23.172807 17889 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33709
I20260924 04:12:23.173044 17889 heartbeater.cc:464] Registering TS with master...
I20260924 04:12:23.173506 17889 heartbeater.cc:511] Master 127.0.98.62:33709 requested a full tablet report, sending...
I20260924 04:12:23.174501 17430 ts_manager.cc:194] Registered new tserver with Master: 5691717cbc9f4ee5ad601d6dcf1cb931 (127.0.98.3:42745)
I20260924 04:12:23.175064 17430 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52439
I20260924 04:12:23.180413 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:12:23.189662 17469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:23.182395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48784 (local address 127.0.98.62:33709)
0924 04:12:23.182544 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:23.182550 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:23.182752 (+ 202us) server_negotiation.cc:407] Connection header received
0924 04:12:23.182894 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:23.182899 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:23.182943 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:23.183035 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:12:23.183720 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:23.184234 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:23.185138 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:23.185375 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:23.185480 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:23.186753 (+ 1273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:12:23.186757 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:12:23.186760 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:12:23.186784 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:12:23.187984 (+ 1200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:12:23.188400 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:12:23.188405 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:12:23.188406 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:12:23.188492 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:12:23.188793 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:12:23.188797 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:12:23.188799 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:12:23.189017 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:12:23.189168 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:23.189386 (+ 218us) server_negotiation.cc:299] Negotiation successful
0924 04:12:23.189505 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 04:12:23.192251 17430 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48784:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:12:23.195279 17430 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:12:23.209738 17900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:23.203442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42745 (local address 127.0.0.1:38094)
0924 04:12:23.204411 (+ 969us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:23.204457 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 04:12:23.204579 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:12:23.205620 (+ 1041us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:12:23.205623 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:12:23.205636 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:12:23.205754 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:23.205759 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:23.206792 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:23.206795 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:23.208057 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:23.208066 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:23.209081 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:23.209084 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:23.209227 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:23.209272 (+ 45us) client_negotiation.cc:769] Sending connection context
0924 04:12:23.209374 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 04:12:23.209534 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":805,"mutex_wait_us":124,"thread_start_us":87,"threads_started":1}
I20260924 04:12:23.209934 17901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:23.203570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38094 (local address 127.0.98.3:42745)
0924 04:12:23.205178 (+ 1608us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:23.205190 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:23.205211 (+ 21us) server_negotiation.cc:407] Connection header received
0924 04:12:23.205278 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:23.205283 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:23.205482 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:23.205859 (+ 377us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:12:23.205972 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:23.206647 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:23.208176 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:23.208955 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:23.209065 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:23.209211 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:23.209630 (+ 419us) server_negotiation.cc:299] Negotiation successful
0924 04:12:23.209756 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1501,"thread_start_us":152,"threads_started":1}
I20260924 04:12:23.209797 17903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:23.203909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52524 (local address 127.0.98.1:39695)
0924 04:12:23.204190 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:23.204207 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:23.204555 (+ 348us) server_negotiation.cc:407] Connection header received
0924 04:12:23.204811 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:23.204818 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:23.204992 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:23.205157 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:12:23.205480 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:23.206675 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:23.208466 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:23.209151 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:23.209251 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:23.209386 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:23.209526 (+ 140us) server_negotiation.cc:299] Negotiation successful
0924 04:12:23.209629 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":108,"threads_started":1}
I20260924 04:12:23.210332 17904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:23.203716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39695 (local address 127.0.0.1:52524)
0924 04:12:23.204409 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:23.204457 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 04:12:23.204630 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:12:23.205211 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:12:23.205216 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:12:23.205241 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:12:23.205360 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:23.205365 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:23.206826 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:23.206832 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:23.208317 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:23.208328 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:23.209293 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:23.209299 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:23.209387 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:23.209408 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 04:12:23.209482 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 04:12:23.209559 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"mutex_wait_us":136,"thread_start_us":83,"threads_started":1}
I20260924 04:12:23.210384 17905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:23.204568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42038 (local address 127.0.98.2:40215)
0924 04:12:23.204960 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:23.204972 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:23.205090 (+ 118us) server_negotiation.cc:407] Connection header received
0924 04:12:23.205172 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:23.205176 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:23.205349 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:23.205508 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:12:23.205704 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:23.206784 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:23.208442 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:23.209251 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:23.209349 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:23.209484 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:23.210120 (+ 636us) server_negotiation.cc:299] Negotiation successful
0924 04:12:23.210232 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":134,"threads_started":1}
I20260924 04:12:23.210675 17902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:23.204141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40215 (local address 127.0.0.1:42038)
0924 04:12:23.204966 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:23.204980 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:12:23.205076 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:12:23.205491 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:12:23.205497 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:12:23.205510 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:12:23.205619 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:23.205625 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:23.206913 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:23.206915 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:23.208317 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:23.208328 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:23.209539 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:23.209542 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:23.209623 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:23.209643 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 04:12:23.209706 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 04:12:23.210531 (+ 825us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":773,"mutex_wait_us":80,"thread_start_us":105,"threads_started":1}
I20260924 04:12:23.213095 17555 tablet_service.cc:1467] Processing CreateTablet for tablet 8c6d67b0de564436996b8a4392f883b9 (DEFAULT_TABLE table=test-table [id=3b61f979d5c1486ebc62d30d1dc99e5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:12:23.212965 17823 tablet_service.cc:1467] Processing CreateTablet for tablet 8c6d67b0de564436996b8a4392f883b9 (DEFAULT_TABLE table=test-table [id=3b61f979d5c1486ebc62d30d1dc99e5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:12:23.214138 17555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c6d67b0de564436996b8a4392f883b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:12:23.214303 17823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c6d67b0de564436996b8a4392f883b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:12:23.215472 17689 tablet_service.cc:1467] Processing CreateTablet for tablet 8c6d67b0de564436996b8a4392f883b9 (DEFAULT_TABLE table=test-table [id=3b61f979d5c1486ebc62d30d1dc99e5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:12:23.216459 17689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c6d67b0de564436996b8a4392f883b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:12:23.220441 17906 tablet_bootstrap.cc:492] T 8c6d67b0de564436996b8a4392f883b9 P 5691717cbc9f4ee5ad601d6dcf1cb931: Bootstrap starting.
I20260924 04:12:23.221480 17907 tablet_bootstrap.cc:492] T 8c6d67b0de564436996b8a4392f883b9 P ff5f2ef59b7d46488f6ce346e80120ac: Bootstrap starting.
I20260924 04:12:23.222613 17906 tablet_bootstrap.cc:654] T 8c6d67b0de564436996b8a4392f883b9 P 5691717cbc9f4ee5ad601d6dcf1cb931: Neither blocks nor log segments found. Creating new log.
I20260924 04:12:23.223636 17907 tablet_bootstrap.cc:654] T 8c6d67b0de564436996b8a4392f883b9 P ff5f2ef59b7d46488f6ce346e80120ac: Neither blocks nor log segments found. Creating new log.
I20260924 04:12:23.223663 17908 tablet_bootstrap.cc:492] T 8c6d67b0de564436996b8a4392f883b9 P 13b187a103584682ab31e9a1119dbf45: Bootstrap starting.
I20260924 04:12:23.223794 17906 log.cc:824] T 8c6d67b0de564436996b8a4392f883b9 P 5691717cbc9f4ee5ad601d6dcf1cb931: Log is configured to *not* fsync() on all Append() calls
I20260924 04:12:23.224529 17907 log.cc:824] T 8c6d67b0de564436996b8a4392f883b9 P ff5f2ef59b7d46488f6ce346e80120ac: Log is configured to *not* fsync() on all Append() calls
I20260924 04:12:23.225914 17908 tablet_bootstrap.cc:654] T 8c6d67b0de564436996b8a4392f883b9 P 13b187a103584682ab31e9a1119dbf45: Neither blocks nor log segments found. Creating new log.
I20260924 04:12:23.226230 17906 tablet_bootstrap.cc:492] T 8c6d67b0de564436996b8a4392f883b9 P 5691717cbc9f4ee5ad601d6dcf1cb931: No bootstrap required, opened a new log
I20260924 04:12:23.226450 17906 ts_tablet_manager.cc:1397] T 8c6d67b0de564436996b8a4392f883b9 P 5691717cbc9f4ee5ad601d6dcf1cb931: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 04:12:23.227177 17908 log.cc:824] T 8c6d67b0de564436996b8a4392f883b9 P 13b187a103584682ab31e9a1119dbf45: Log is configured to *not* fsync() on all Append() calls
I20260924 04:12:23.227415 17907 tablet_bootstrap.cc:492] T 8c6d67b0de564436996b8a4392f883b9 P ff5f2ef59b7d46488f6ce346e80120ac: No bootstrap required, opened a new log
I20260924 04:12:23.227605 17907 ts_tablet_manager.cc:1397] T 8c6d67b0de564436996b8a4392f883b9 P ff5f2ef59b7d46488f6ce346e80120ac: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 04:12:23.229080 17908 tablet_bootstrap.cc:492] T 8c6d67b0de564436996b8a4392f883b9 P 13b187a103584682ab31e9a1119dbf45: No bootstrap required, opened a new log
I20260924 04:12:23.229266 17908 ts_tablet_manager.cc:1397] T 8c6d67b0de564436996b8a4392f883b9 P 13b187a103584682ab31e9a1119dbf45: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 04:12:23.230341 17906 raft_consensus.cc:348] T 8c6d67b0de564436996b8a4392f883b9 P 5691717cbc9f4ee5ad601d6dcf1cb931 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5691717cbc9f4ee5ad601d6dcf1cb931" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42745 } } peers { permanent_uuid: "13b187a103584682ab31e9a1119dbf45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40215 } } peers { permanent_uuid: "ff5f2ef59b7d46488f6ce346e80120ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39695 } }
I20260924 04:12:23.230630 17906 raft_consensus.cc:374] T 8c6d67b0de564436996b8a4392f883b9 P 5691717cbc9f4ee5ad601d6dcf1cb931 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:12:23.230720 17906 raft_consensus.cc:729] T 8c6d67b0de564436996b8a4392f883b9 P 5691717cbc9f4ee5ad601d6dcf1cb931 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5691717cbc9f4ee5ad601d6dcf1cb931, State: Initialized, Role: FOLLOWER
I20260924 04:12:23.231034 17907 raft_consensus.cc:348] T 8c6d67b0de564436996b8a4392f883b9 P ff5f2ef59b7d46488f6ce346e80120ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5691717cbc9f4ee5ad601d6dcf1cb931" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42745 } } peers { permanent_uuid: "13b187a103584682ab31e9a1119dbf45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40215 } } peers { permanent_uuid: "ff5f2ef59b7d46488f6ce346e80120ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39695 } }
I20260924 04:12:23.231213 17906 consensus_queue.cc:260] T 8c6d67b0de564436996b8a4392f883b9 P 5691717cbc9f4ee5ad601d6dcf1cb931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5691717cbc9f4ee5ad601d6dcf1cb931" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42745 } } peers { permanent_uuid: "13b187a103584682ab31e9a1119dbf45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40215 } } peers { permanent_uuid: "ff5f2ef59b7d46488f6ce346e80120ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39695 } }
I20260924 04:12:23.231467 17907 raft_consensus.cc:374] T 8c6d67b0de564436996b8a4392f883b9 P ff5f2ef59b7d46488f6ce346e80120ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:12:23.231578 17907 raft_consensus.cc:729] T 8c6d67b0de564436996b8a4392f883b9 P ff5f2ef59b7d46488f6ce346e80120ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5f2ef59b7d46488f6ce346e80120ac, State: Initialized, Role: FOLLOWER
I20260924 04:12:23.232045 17907 consensus_queue.cc:260] T 8c6d67b0de564436996b8a4392f883b9 P ff5f2ef59b7d46488f6ce346e80120ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5691717cbc9f4ee5ad601d6dcf1cb931" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42745 } } peers { permanent_uuid: "13b187a103584682ab31e9a1119dbf45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40215 } } peers { permanent_uuid: "ff5f2ef59b7d46488f6ce346e80120ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39695 } }
I20260924 04:12:23.232189 17906 ts_tablet_manager.cc:1428] T 8c6d67b0de564436996b8a4392f883b9 P 5691717cbc9f4ee5ad601d6dcf1cb931: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260924 04:12:23.232542 17889 heartbeater.cc:503] Master 127.0.98.62:33709 was elected leader, sending a full tablet report...
I20260924 04:12:23.232668 17621 heartbeater.cc:503] Master 127.0.98.62:33709 was elected leader, sending a full tablet report...
I20260924 04:12:23.233260 17908 raft_consensus.cc:348] T 8c6d67b0de564436996b8a4392f883b9 P 13b187a103584682ab31e9a1119dbf45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5691717cbc9f4ee5ad601d6dcf1cb931" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42745 } } peers { permanent_uuid: "13b187a103584682ab31e9a1119dbf45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40215 } } peers { permanent_uuid: "ff5f2ef59b7d46488f6ce346e80120ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39695 } }
I20260924 04:12:23.233583 17908 raft_consensus.cc:374] T 8c6d67b0de564436996b8a4392f883b9 P 13b187a103584682ab31e9a1119dbf45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:12:23.233685 17908 raft_consensus.cc:729] T 8c6d67b0de564436996b8a4392f883b9 P 13b187a103584682ab31e9a1119dbf45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13b187a103584682ab31e9a1119dbf45, State: Initialized, Role: FOLLOWER
I20260924 04:12:23.233834 17907 ts_tablet_manager.cc:1428] T 8c6d67b0de564436996b8a4392f883b9 P ff5f2ef59b7d46488f6ce346e80120ac: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 04:12:23.234160 17908 consensus_queue.cc:260] T 8c6d67b0de564436996b8a4392f883b9 P 13b187a103584682ab31e9a1119dbf45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5691717cbc9f4ee5ad601d6dcf1cb931" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42745 } } peers { permanent_uuid: "13b187a103584682ab31e9a1119dbf45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40215 } } peers { permanent_uuid: "ff5f2ef59b7d46488f6ce346e80120ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39695 } }
I20260924 04:12:23.234750 17755 heartbeater.cc:503] Master 127.0.98.62:33709 was elected leader, sending a full tablet report...
I20260924 04:12:23.235038 17908 ts_tablet_manager.cc:1428] T 8c6d67b0de564436996b8a4392f883b9 P 13b187a103584682ab31e9a1119dbf45: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260924 04:12:23.323659 17912 raft_consensus.cc:482] T 8c6d67b0de564436996b8a4392f883b9 P 5691717cbc9f4ee5ad601d6dcf1cb931 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:12:23.323904 17912 raft_consensus.cc:504] T 8c6d67b0de564436996b8a4392f883b9 P 5691717cbc9f4ee5ad601d6dcf1cb931 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5691717cbc9f4ee5ad601d6dcf1cb931" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42745 } } peers { permanent_uuid: "13b187a103584682ab31e9a1119dbf45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40215 } } peers { permanent_uuid: "ff5f2ef59b7d46488f6ce346e80120ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39695 } }
I20260924 04:12:23.325253 17912 leader_election.cc:288] T 8c6d67b0de564436996b8a4392f883b9 P 5691717cbc9f4ee5ad601d6dcf1cb931 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 13b187a103584682ab31e9a1119dbf45 (127.0.98.2:40215), ff5f2ef59b7d46488f6ce346e80120ac (127.0.98.1:39695)
I20260924 04:12:23.330320 17905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:23.325646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35241 (local address 127.0.98.2:40215)
0924 04:12:23.325777 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:23.325783 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:23.325797 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:12:23.325850 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:23.325853 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:23.325904 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:23.326012 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:12:23.326331 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:23.327112 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:23.328817 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:23.329447 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:23.329556 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:23.329623 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:23.330105 (+ 482us) server_negotiation.cc:299] Negotiation successful
0924 04:12:23.330194 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 04:12:23.330320 17903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:23.325694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52749 (local address 127.0.98.1:39695)
0924 04:12:23.325870 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:23.325875 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:23.325968 (+ 93us) server_negotiation.cc:407] Connection header received
0924 04:12:23.326144 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:23.326153 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:23.326193 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:23.326249 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:12:23.326506 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:23.327214 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:23.328817 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:23.329447 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:23.329556 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:23.329623 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:23.330105 (+ 482us) server_negotiation.cc:299] Negotiation successful
0924 04:12:23.330184 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260924 04:12:23.330343 17891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:23.325439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40215 (local address 127.0.98.3:35241)
0924 04:12:23.325584 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:23.325601 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 04:12:23.325691 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:12:23.326043 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:12:23.326046 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:12:23.326062 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:12:23.326196 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:23.326201 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:23.327229 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:23.327231 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:23.328667 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:23.328698 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:23.329592 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:23.329596 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:23.329870 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:23.329985 (+ 115us) client_negotiation.cc:769] Sending connection context
0924 04:12:23.330071 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 04:12:23.330183 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58,"mutex_wait_us":184}
I20260924 04:12:23.330353 17915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:23.325523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39695 (local address 127.0.98.3:52749)
0924 04:12:23.325873 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:23.325885 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:12:23.325972 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:12:23.326294 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:12:23.326298 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:12:23.326318 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:12:23.326408 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:23.326412 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:23.327326 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:23.327329 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:23.328663 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:23.328699 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:23.329592 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:23.329596 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:23.329868 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:23.329985 (+ 117us) client_negotiation.cc:769] Sending connection context
0924 04:12:23.330078 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 04:12:23.330183 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"mutex_wait_us":218,"thread_start_us":124,"threads_started":1}
I20260924 04:12:23.331072 17575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c6d67b0de564436996b8a4392f883b9" candidate_uuid: "5691717cbc9f4ee5ad601d6dcf1cb931" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5f2ef59b7d46488f6ce346e80120ac" is_pre_election: true
I20260924 04:12:23.331074 17709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c6d67b0de564436996b8a4392f883b9" candidate_uuid: "5691717cbc9f4ee5ad601d6dcf1cb931" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13b187a103584682ab31e9a1119dbf45" is_pre_election: true
I20260924 04:12:23.331370 17709 raft_consensus.cc:2454] T 8c6d67b0de564436996b8a4392f883b9 P 13b187a103584682ab31e9a1119dbf45 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5691717cbc9f4ee5ad601d6dcf1cb931 in term 0.
I20260924 04:12:23.331379 17575 raft_consensus.cc:2454] T 8c6d67b0de564436996b8a4392f883b9 P ff5f2ef59b7d46488f6ce346e80120ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5691717cbc9f4ee5ad601d6dcf1cb931 in term 0.
I20260924 04:12:23.331889 17776 leader_election.cc:302] T 8c6d67b0de564436996b8a4392f883b9 P 5691717cbc9f4ee5ad601d6dcf1cb931 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 13b187a103584682ab31e9a1119dbf45, 5691717cbc9f4ee5ad601d6dcf1cb931, ff5f2ef59b7d46488f6ce346e80120ac; no voters:
I20260924 04:12:23.332202 17912 raft_consensus.cc:2781] T 8c6d67b0de564436996b8a4392f883b9 P 5691717cbc9f4ee5ad601d6dcf1cb931 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:12:23.332280 17912 raft_consensus.cc:482] T 8c6d67b0de564436996b8a4392f883b9 P 5691717cbc9f4ee5ad601d6dcf1cb931 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:12:23.332333 17912 raft_consensus.cc:3037] T 8c6d67b0de564436996b8a4392f883b9 P 5691717cbc9f4ee5ad601d6dcf1cb931 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:12:23.333379 17912 raft_consensus.cc:504] T 8c6d67b0de564436996b8a4392f883b9 P 5691717cbc9f4ee5ad601d6dcf1cb931 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5691717cbc9f4ee5ad601d6dcf1cb931" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42745 } } peers { permanent_uuid: "13b187a103584682ab31e9a1119dbf45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40215 } } peers { permanent_uuid: "ff5f2ef59b7d46488f6ce346e80120ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39695 } }
I20260924 04:12:23.333783 17912 leader_election.cc:288] T 8c6d67b0de564436996b8a4392f883b9 P 5691717cbc9f4ee5ad601d6dcf1cb931 [CANDIDATE]: Term 1 election: Requested vote from peers 13b187a103584682ab31e9a1119dbf45 (127.0.98.2:40215), ff5f2ef59b7d46488f6ce346e80120ac (127.0.98.1:39695)
I20260924 04:12:23.334064 17709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c6d67b0de564436996b8a4392f883b9" candidate_uuid: "5691717cbc9f4ee5ad601d6dcf1cb931" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13b187a103584682ab31e9a1119dbf45"
I20260924 04:12:23.334159 17575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c6d67b0de564436996b8a4392f883b9" candidate_uuid: "5691717cbc9f4ee5ad601d6dcf1cb931" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5f2ef59b7d46488f6ce346e80120ac"
I20260924 04:12:23.334213 17709 raft_consensus.cc:3037] T 8c6d67b0de564436996b8a4392f883b9 P 13b187a103584682ab31e9a1119dbf45 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:12:23.334295 17575 raft_consensus.cc:3037] T 8c6d67b0de564436996b8a4392f883b9 P ff5f2ef59b7d46488f6ce346e80120ac [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:12:23.335287 17575 raft_consensus.cc:2454] T 8c6d67b0de564436996b8a4392f883b9 P ff5f2ef59b7d46488f6ce346e80120ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5691717cbc9f4ee5ad601d6dcf1cb931 in term 1.
I20260924 04:12:23.335359 17709 raft_consensus.cc:2454] T 8c6d67b0de564436996b8a4392f883b9 P 13b187a103584682ab31e9a1119dbf45 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5691717cbc9f4ee5ad601d6dcf1cb931 in term 1.
I20260924 04:12:23.335553 17776 leader_election.cc:302] T 8c6d67b0de564436996b8a4392f883b9 P 5691717cbc9f4ee5ad601d6dcf1cb931 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5691717cbc9f4ee5ad601d6dcf1cb931, ff5f2ef59b7d46488f6ce346e80120ac; no voters:
I20260924 04:12:23.335765 17912 raft_consensus.cc:2781] T 8c6d67b0de564436996b8a4392f883b9 P 5691717cbc9f4ee5ad601d6dcf1cb931 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:12:23.335940 17912 raft_consensus.cc:686] T 8c6d67b0de564436996b8a4392f883b9 P 5691717cbc9f4ee5ad601d6dcf1cb931 [term 1 LEADER]: Becoming Leader. State: Replica: 5691717cbc9f4ee5ad601d6dcf1cb931, State: Running, Role: LEADER
I20260924 04:12:23.336275 17912 consensus_queue.cc:237] T 8c6d67b0de564436996b8a4392f883b9 P 5691717cbc9f4ee5ad601d6dcf1cb931 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5691717cbc9f4ee5ad601d6dcf1cb931" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42745 } } peers { permanent_uuid: "13b187a103584682ab31e9a1119dbf45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40215 } } peers { permanent_uuid: "ff5f2ef59b7d46488f6ce346e80120ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39695 } }
I20260924 04:12:23.339731 17430 catalog_manager.cc:5495] T 8c6d67b0de564436996b8a4392f883b9 P 5691717cbc9f4ee5ad601d6dcf1cb931 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5691717cbc9f4ee5ad601d6dcf1cb931 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "5691717cbc9f4ee5ad601d6dcf1cb931" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5691717cbc9f4ee5ad601d6dcf1cb931" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13b187a103584682ab31e9a1119dbf45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff5f2ef59b7d46488f6ce346e80120ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39695 } health_report { overall_health: UNKNOWN } } }
I20260924 04:12:23.360117 17901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:23.356960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38100 (local address 127.0.98.3:42745)
0924 04:12:23.357090 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:23.357094 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:23.357106 (+ 12us) server_negotiation.cc:407] Connection header received
0924 04:12:23.357198 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:23.357200 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:23.357253 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:23.357342 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:12:23.357530 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:23.358023 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:23.358824 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:23.359023 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:23.359094 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:23.359361 (+ 267us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:12:23.359776 (+ 415us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:12:23.359865 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:23.359947 (+ 82us) server_negotiation.cc:299] Negotiation successful
0924 04:12:23.359994 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 04:12:23.365916 17709 raft_consensus.cc:1261] T 8c6d67b0de564436996b8a4392f883b9 P 13b187a103584682ab31e9a1119dbf45 [term 1 FOLLOWER]: Refusing update from remote peer 5691717cbc9f4ee5ad601d6dcf1cb931: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:12:23.365939 17575 raft_consensus.cc:1261] T 8c6d67b0de564436996b8a4392f883b9 P ff5f2ef59b7d46488f6ce346e80120ac [term 1 FOLLOWER]: Refusing update from remote peer 5691717cbc9f4ee5ad601d6dcf1cb931: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:12:23.366623 17912 consensus_queue.cc:1035] T 8c6d67b0de564436996b8a4392f883b9 P 5691717cbc9f4ee5ad601d6dcf1cb931 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13b187a103584682ab31e9a1119dbf45" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:12:23.366827 17916 consensus_queue.cc:1035] T 8c6d67b0de564436996b8a4392f883b9 P 5691717cbc9f4ee5ad601d6dcf1cb931 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff5f2ef59b7d46488f6ce346e80120ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:12:23.372646 17919 mvcc.cc:205] Tried to move back new op lower bound from 7332753995219652608 to 7332753995108466688. Current Snapshot: MvccSnapshot[applied={T|T < 7332753995219652608}]
I20260924 04:12:23.383955 17430 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48784:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:12:23.384141 17430 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48784:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:12:23.386687 17430 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a07ac4d42fc54e93ab69ff8b39bcd60d: Sending DeleteTablet for 3 replicas of tablet 8c6d67b0de564436996b8a4392f883b9
I20260924 04:12:23.387460 17689 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c6d67b0de564436996b8a4392f883b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:12:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42038
I20260924 04:12:23.387939 17929 tablet_replica.cc:330] T 8c6d67b0de564436996b8a4392f883b9 P 13b187a103584682ab31e9a1119dbf45: stopping tablet replica
I20260924 04:12:23.387921 17555 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c6d67b0de564436996b8a4392f883b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:12:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52524
I20260924 04:12:23.388365 17929 raft_consensus.cc:2229] T 8c6d67b0de564436996b8a4392f883b9 P 13b187a103584682ab31e9a1119dbf45 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:12:23.388664 17823 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c6d67b0de564436996b8a4392f883b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:12:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38094
I20260924 04:12:23.388783 17929 raft_consensus.cc:2258] T 8c6d67b0de564436996b8a4392f883b9 P 13b187a103584682ab31e9a1119dbf45 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:12:23.389591 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 17492
I20260924 04:12:23.394205 17929 ts_tablet_manager.cc:1905] T 8c6d67b0de564436996b8a4392f883b9 P 13b187a103584682ab31e9a1119dbf45: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:12:23.397079 17929 ts_tablet_manager.cc:1918] T 8c6d67b0de564436996b8a4392f883b9 P 13b187a103584682ab31e9a1119dbf45: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:12:23.397253 17929 log.cc:1196] T 8c6d67b0de564436996b8a4392f883b9 P 13b187a103584682ab31e9a1119dbf45: Deleting WAL directory at /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222898807720-392-0/minicluster-data/ts-1/wal/wals/8c6d67b0de564436996b8a4392f883b9
I20260924 04:12:23.397660 17929 ts_tablet_manager.cc:1939] T 8c6d67b0de564436996b8a4392f883b9 P 13b187a103584682ab31e9a1119dbf45: Deleting consensus metadata
W20260924 04:12:23.398921 17414 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39695 (error 108)
I20260924 04:12:23.398926 17416 catalog_manager.cc:4841] TS 13b187a103584682ab31e9a1119dbf45 (127.0.98.2:40215): tablet 8c6d67b0de564436996b8a4392f883b9 (table test-table [id=3b61f979d5c1486ebc62d30d1dc99e5b]) successfully deleted
I20260924 04:12:23.399858 17902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:23.399622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39695 (local address 127.0.0.1:52538)
0924 04:12:23.399729 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:23.399789 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260924 04:12:23.400051 17414 catalog_manager.cc:4584] TS ff5f2ef59b7d46488f6ce346e80120ac (127.0.98.1:39695): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8c6d67b0de564436996b8a4392f883b9: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39695: connect: Connection refused (error 111)
I20260924 04:12:23.400172 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 17626
I20260924 04:12:23.400913 17931 tablet_replica.cc:330] T 8c6d67b0de564436996b8a4392f883b9 P 5691717cbc9f4ee5ad601d6dcf1cb931: stopping tablet replica
I20260924 04:12:23.401247 17931 raft_consensus.cc:2229] T 8c6d67b0de564436996b8a4392f883b9 P 5691717cbc9f4ee5ad601d6dcf1cb931 [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:12:23.401660 17931 raft_consensus.cc:2258] T 8c6d67b0de564436996b8a4392f883b9 P 5691717cbc9f4ee5ad601d6dcf1cb931 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:12:23.405848 17931 ts_tablet_manager.cc:1905] T 8c6d67b0de564436996b8a4392f883b9 P 5691717cbc9f4ee5ad601d6dcf1cb931: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:12:23.408597 17931 ts_tablet_manager.cc:1918] T 8c6d67b0de564436996b8a4392f883b9 P 5691717cbc9f4ee5ad601d6dcf1cb931: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:12:23.408778 17931 log.cc:1196] T 8c6d67b0de564436996b8a4392f883b9 P 5691717cbc9f4ee5ad601d6dcf1cb931: Deleting WAL directory at /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790222898807720-392-0/minicluster-data/ts-2/wal/wals/8c6d67b0de564436996b8a4392f883b9
I20260924 04:12:23.408931 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 17760
I20260924 04:12:23.409173 17931 ts_tablet_manager.cc:1939] T 8c6d67b0de564436996b8a4392f883b9 P 5691717cbc9f4ee5ad601d6dcf1cb931: Deleting consensus metadata
I20260924 04:12:23.414593 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 17398
2026-09-24T04:12:23Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2392 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-24T04:12:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:12:23Z Disabled control of system clock
I20260924 04:12:23.444816 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222898807720-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222898807720-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222898807720-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40763
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:35747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40763
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 04:12:23.554652 17936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:12:23.554940 17936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:12:23.554978 17936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:12:23.558966 17936 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:12:23.559032 17936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:12:23.559048 17936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:12:23.559065 17936 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:12:23.559082 17936 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:12:23.563905 17936 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222898807720-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222898807720-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40763
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40763
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17936
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222898807720-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:12:23.564970 17936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:12:23.566315 17936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:12:23.574390 17941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:12:23.574398 17944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:12:23.574638 17936 server_base.cc:1034] running on GCE node
W20260924 04:12:23.574402 17942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:12:23.575217 17936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:12:23.576233 17936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:12:23.577383 17936 hybrid_clock.cc:648] HybridClock initialized: now 1790223143577352 us; error 44 us; skew 500 ppm
I20260924 04:12:23.579092 17936 webserver.cc:460] Webserver started at http://127.0.98.62:45519/ using document root <none> and password file <none>
I20260924 04:12:23.579622 17936 fs_manager.cc:359] Metadata directory not provided
I20260924 04:12:23.579677 17936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:12:23.579844 17936 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:12:23.581813 17936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222898807720-392-0/minicluster-data/master-0/data/instance:
uuid: "fb19d1bcb9274367802d30380e193c47"
format_stamp: "Formatted at 2026-09-24 04:12:23 on dist-test-slave-00ld"
server_key: "bf7ee22a1cc9f70c1133c0b8af3f1997"
server_key_iv: "6e1a7a1fb104a6c4af3d905dc368fa5d"
server_key_version: "encryptionkey@0"
I20260924 04:12:23.582297 17936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222898807720-392-0/minicluster-data/master-0/wal/instance:
uuid: "fb19d1bcb9274367802d30380e193c47"
format_stamp: "Formatted at 2026-09-24 04:12:23 on dist-test-slave-00ld"
server_key: "bf7ee22a1cc9f70c1133c0b8af3f1997"
server_key_iv: "6e1a7a1fb104a6c4af3d905dc368fa5d"
server_key_version: "encryptionkey@0"
I20260924 04:12:23.585600 17936 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:12:23.587800 17950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:12:23.589080 17936 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 04:12:23.589221 17936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222898807720-392-0/minicluster-data/master-0/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222898807720-392-0/minicluster-data/master-0/wal
uuid: "fb19d1bcb9274367802d30380e193c47"
format_stamp: "Formatted at 2026-09-24 04:12:23 on dist-test-slave-00ld"
server_key: "bf7ee22a1cc9f70c1133c0b8af3f1997"
server_key_iv: "6e1a7a1fb104a6c4af3d905dc368fa5d"
server_key_version: "encryptionkey@0"
I20260924 04:12:23.589326 17936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222898807720-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222898807720-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222898807720-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:12:23.598716 17936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:12:23.599319 17936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:12:23.599499 17936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:12:23.606197 17936 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40763
I20260924 04:12:23.606213 18002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40763 every 8 connection(s)
I20260924 04:12:23.607115 17936 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:12:23.610086 18003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:12:23.610260 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 17936
I20260924 04:12:23.610399 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222898807720-392-0/minicluster-data/master-0/wal/instance
I20260924 04:12:23.610687 392 external_mini_cluster.cc:1428] Setting key 9554c80036e3dd263b19ea92851533bd
I20260924 04:12:23.617120 18003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb19d1bcb9274367802d30380e193c47: Bootstrap starting.
I20260924 04:12:23.619213 18003 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb19d1bcb9274367802d30380e193c47: Neither blocks nor log segments found. Creating new log.
I20260924 04:12:23.620031 18003 log.cc:824] T 00000000000000000000000000000000 P fb19d1bcb9274367802d30380e193c47: Log is configured to *not* fsync() on all Append() calls
I20260924 04:12:23.620363 18006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:23.612071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33660 (local address 127.0.98.62:40763)
0924 04:12:23.612751 (+ 680us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:23.612770 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:23.612811 (+ 41us) server_negotiation.cc:407] Connection header received
0924 04:12:23.613534 (+ 723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:23.613556 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:23.614058 (+ 502us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:23.614429 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:12:23.614857 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:23.616368 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:23.617270 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:23.617740 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:23.617917 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:23.618736 (+ 819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:12:23.618740 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:12:23.618752 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:12:23.619112 (+ 360us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:12:23.619162 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:12:23.619181 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:12:23.619286 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:23.619414 (+ 128us) server_negotiation.cc:299] Negotiation successful
0924 04:12:23.619655 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":112,"threads_started":1}
I20260924 04:12:23.621968 18003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb19d1bcb9274367802d30380e193c47: No bootstrap required, opened a new log
I20260924 04:12:23.625600 18003 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb19d1bcb9274367802d30380e193c47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb19d1bcb9274367802d30380e193c47" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40763 } }
I20260924 04:12:23.625833 18003 raft_consensus.cc:374] T 00000000000000000000000000000000 P fb19d1bcb9274367802d30380e193c47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:12:23.625921 18003 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb19d1bcb9274367802d30380e193c47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb19d1bcb9274367802d30380e193c47, State: Initialized, Role: FOLLOWER
I20260924 04:12:23.626323 18003 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb19d1bcb9274367802d30380e193c47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb19d1bcb9274367802d30380e193c47" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40763 } }
I20260924 04:12:23.626464 18003 raft_consensus.cc:388] T 00000000000000000000000000000000 P fb19d1bcb9274367802d30380e193c47 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:12:23.626549 18003 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb19d1bcb9274367802d30380e193c47 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:12:23.626663 18003 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb19d1bcb9274367802d30380e193c47 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:12:23.627522 18003 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb19d1bcb9274367802d30380e193c47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb19d1bcb9274367802d30380e193c47" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40763 } }
I20260924 04:12:23.627858 18003 leader_election.cc:302] T 00000000000000000000000000000000 P fb19d1bcb9274367802d30380e193c47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb19d1bcb9274367802d30380e193c47; no voters:
I20260924 04:12:23.628185 18003 leader_election.cc:288] T 00000000000000000000000000000000 P fb19d1bcb9274367802d30380e193c47 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:12:23.628288 18008 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb19d1bcb9274367802d30380e193c47 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:12:23.628553 18008 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb19d1bcb9274367802d30380e193c47 [term 1 LEADER]: Becoming Leader. State: Replica: fb19d1bcb9274367802d30380e193c47, State: Running, Role: LEADER
I20260924 04:12:23.628897 18008 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb19d1bcb9274367802d30380e193c47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb19d1bcb9274367802d30380e193c47" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40763 } }
I20260924 04:12:23.629400 18003 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb19d1bcb9274367802d30380e193c47 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:12:23.630470 18009 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb19d1bcb9274367802d30380e193c47 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fb19d1bcb9274367802d30380e193c47" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb19d1bcb9274367802d30380e193c47" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40763 } } }
I20260924 04:12:23.630594 18009 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb19d1bcb9274367802d30380e193c47 [sys.catalog]: This master's current role is: LEADER
I20260924 04:12:23.630937 18017 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:12:23.630906 18010 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb19d1bcb9274367802d30380e193c47 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb19d1bcb9274367802d30380e193c47. Latest consensus state: current_term: 1 leader_uuid: "fb19d1bcb9274367802d30380e193c47" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb19d1bcb9274367802d30380e193c47" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40763 } } }
I20260924 04:12:23.631142 18010 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb19d1bcb9274367802d30380e193c47 [sys.catalog]: This master's current role is: LEADER
I20260924 04:12:23.633780 18017 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:12:23.639397 18017 catalog_manager.cc:1347] Generated new cluster ID: 33c422bd9adb42c8b221a1432f9c2281
I20260924 04:12:23.639549 18017 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:12:23.658181 18017 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:12:23.659159 18017 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:12:23.670565 18017 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fb19d1bcb9274367802d30380e193c47: Generated new TSK 0
I20260924 04:12:23.671329 18017 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:12:23.681373 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222898807720-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222898807720-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222898807720-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222898807720-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40763
--builtin_ntp_servers=127.0.98.20:35747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 04:12:23.796423 18027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:12:23.796808 18027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:12:23.796885 18027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:12:23.801134 18027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:12:23.801198 18027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:12:23.801285 18027 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:12:23.806386 18027 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222898807720-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222898807720-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222898807720-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40763
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18027
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222898807720-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:12:23.807590 18027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:12:23.809043 18027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:12:23.819037 18035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:12:23.818975 18033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:12:23.819025 18027 server_base.cc:1034] running on GCE node
W20260924 04:12:23.819181 18032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:12:23.819674 18027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:12:23.820369 18027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:12:23.821549 18027 hybrid_clock.cc:648] HybridClock initialized: now 1790223143821532 us; error 30 us; skew 500 ppm
I20260924 04:12:23.823356 18027 webserver.cc:460] Webserver started at http://127.0.98.1:36947/ using document root <none> and password file <none>
I20260924 04:12:23.823932 18027 fs_manager.cc:359] Metadata directory not provided
I20260924 04:12:23.824023 18027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:12:23.824241 18027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:12:23.826333 18027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222898807720-392-0/minicluster-data/ts-0/data/instance:
uuid: "b2e26e21d980479caee07e53c16982ed"
format_stamp: "Formatted at 2026-09-24 04:12:23 on dist-test-slave-00ld"
server_key: "6b10cd242bc5e719a6e1e1583e8d6a8b"
server_key_iv: "c214184c98d5d234cd224b594f21d105"
server_key_version: "encryptionkey@0"
I20260924 04:12:23.826929 18027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222898807720-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b2e26e21d980479caee07e53c16982ed"
format_stamp: "Formatted at 2026-09-24 04:12:23 on dist-test-slave-00ld"
server_key: "6b10cd242bc5e719a6e1e1583e8d6a8b"
server_key_iv: "c214184c98d5d234cd224b594f21d105"
server_key_version: "encryptionkey@0"
I20260924 04:12:23.830852 18027 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260924 04:12:23.833384 18041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:12:23.834621 18027 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 04:12:23.834762 18027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222898807720-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222898807720-392-0/minicluster-data/ts-0/wal
uuid: "b2e26e21d980479caee07e53c16982ed"
format_stamp: "Formatted at 2026-09-24 04:12:23 on dist-test-slave-00ld"
server_key: "6b10cd242bc5e719a6e1e1583e8d6a8b"
server_key_iv: "c214184c98d5d234cd224b594f21d105"
server_key_version: "encryptionkey@0"
I20260924 04:12:23.834877 18027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222898807720-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222898807720-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222898807720-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:12:23.854370 18027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:12:23.855113 18027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:12:23.855319 18027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:12:23.855926 18027 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:12:23.856920 18027 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:12:23.856990 18027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:12:23.857062 18027 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:12:23.857105 18027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:12:23.866060 18027 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38537
I20260924 04:12:23.866077 18154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38537 every 8 connection(s)
I20260924 04:12:23.866974 18027 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222898807720-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:12:23.867967 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 18027
I20260924 04:12:23.868108 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222898807720-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:12:23.868448 392 external_mini_cluster.cc:1428] Setting key 413ae70e01efcd338ccbcb7214a740a1
I20260924 04:12:23.870617 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222898807720-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222898807720-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222898807720-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222898807720-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40763
--builtin_ntp_servers=127.0.98.20:35747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 04:12:23.877945 18006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:23.869048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36993 (local address 127.0.98.62:40763)
0924 04:12:23.869189 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:23.869193 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:23.870479 (+ 1286us) server_negotiation.cc:407] Connection header received
0924 04:12:23.871950 (+ 1471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:23.871953 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:23.872009 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:23.872082 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:12:23.873061 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:23.873774 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:23.875696 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:23.875922 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:23.876007 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:23.877464 (+ 1457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:12:23.877467 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:12:23.877469 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:12:23.877482 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:12:23.877499 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:12:23.877507 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:12:23.877561 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:23.877772 (+ 211us) server_negotiation.cc:299] Negotiation successful
0924 04:12:23.877829 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 04:12:23.879238 18157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:23.869559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40763 (local address 127.0.98.1:36993)
0924 04:12:23.870304 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:23.870356 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 04:12:23.871719 (+ 1363us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:12:23.872261 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:12:23.872268 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:12:23.872353 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:12:23.872920 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:23.872935 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:23.873928 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:23.873935 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:23.875552 (+ 1617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:23.875568 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:23.876101 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:23.876107 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:23.876358 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:23.877108 (+ 750us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:12:23.877112 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:12:23.877176 (+ 64us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:12:23.877284 (+ 108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:12:23.877592 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:12:23.877604 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:12:23.877612 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 04:12:23.877760 (+ 148us) client_negotiation.cc:240] Negotiation successful
0924 04:12:23.878098 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":152,"threads_started":1}
I20260924 04:12:23.880564 18155 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40763
I20260924 04:12:23.880834 18155 heartbeater.cc:464] Registering TS with master...
I20260924 04:12:23.881374 18155 heartbeater.cc:511] Master 127.0.98.62:40763 requested a full tablet report, sending...
I20260924 04:12:23.882912 17967 ts_manager.cc:194] Registered new tserver with Master: b2e26e21d980479caee07e53c16982ed (127.0.98.1:38537)
I20260924 04:12:23.884135 17967 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:36993
W20260924 04:12:23.982240 18158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:12:23.982494 18158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:12:23.982532 18158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:12:23.986567 18158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:12:23.986618 18158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:12:23.986696 18158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:12:23.991314 18158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222898807720-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222898807720-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222898807720-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40763
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222898807720-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:12:23.992431 18158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:12:23.993956 18158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:12:24.003892 18164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:12:24.003867 18163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:12:24.003859 18166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:12:24.004210 18158 server_base.cc:1034] running on GCE node
I20260924 04:12:24.004602 18158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:12:24.005273 18158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:12:24.006475 18158 hybrid_clock.cc:648] HybridClock initialized: now 1790223144006466 us; error 31 us; skew 500 ppm
I20260924 04:12:24.008213 18158 webserver.cc:460] Webserver started at http://127.0.98.2:43479/ using document root <none> and password file <none>
I20260924 04:12:24.008875 18158 fs_manager.cc:359] Metadata directory not provided
I20260924 04:12:24.008972 18158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:12:24.009248 18158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:12:24.011303 18158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222898807720-392-0/minicluster-data/ts-1/data/instance:
uuid: "ef46fefdd279418a90e7c85cbfa068be"
format_stamp: "Formatted at 2026-09-24 04:12:24 on dist-test-slave-00ld"
server_key: "f5b2aad67fdb695cf30ae0552ab63382"
server_key_iv: "4a291a7a7dbfcc1efe5e95a54aef2a3c"
server_key_version: "encryptionkey@0"
I20260924 04:12:24.011858 18158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222898807720-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ef46fefdd279418a90e7c85cbfa068be"
format_stamp: "Formatted at 2026-09-24 04:12:24 on dist-test-slave-00ld"
server_key: "f5b2aad67fdb695cf30ae0552ab63382"
server_key_iv: "4a291a7a7dbfcc1efe5e95a54aef2a3c"
server_key_version: "encryptionkey@0"
I20260924 04:12:24.015542 18158 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20260924 04:12:24.018000 18172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:12:24.019196 18158 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:12:24.019318 18158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222898807720-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222898807720-392-0/minicluster-data/ts-1/wal
uuid: "ef46fefdd279418a90e7c85cbfa068be"
format_stamp: "Formatted at 2026-09-24 04:12:24 on dist-test-slave-00ld"
server_key: "f5b2aad67fdb695cf30ae0552ab63382"
server_key_iv: "4a291a7a7dbfcc1efe5e95a54aef2a3c"
server_key_version: "encryptionkey@0"
I20260924 04:12:24.019414 18158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222898807720-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222898807720-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222898807720-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:12:24.030087 18158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:12:24.030751 18158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:12:24.030923 18158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:12:24.031450 18158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:12:24.032294 18158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:12:24.032338 18158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:12:24.032399 18158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:12:24.032442 18158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:12:24.041290 18158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44479
I20260924 04:12:24.041306 18285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44479 every 8 connection(s)
I20260924 04:12:24.042201 18158 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222898807720-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:12:24.046813 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 18158
I20260924 04:12:24.046957 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222898807720-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:12:24.047327 392 external_mini_cluster.cc:1428] Setting key df9880fc55f14376d920ca7f009c19a8
I20260924 04:12:24.049396 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222898807720-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222898807720-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222898807720-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222898807720-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40763
--builtin_ntp_servers=127.0.98.20:35747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 04:12:24.052088 18006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:24.044150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56629 (local address 127.0.98.62:40763)
0924 04:12:24.044324 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:24.044327 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:24.045372 (+ 1045us) server_negotiation.cc:407] Connection header received
0924 04:12:24.046974 (+ 1602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:24.046978 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:24.047041 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:24.047150 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:12:24.048239 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:24.048852 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:24.050107 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:24.050339 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:24.050450 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:24.051585 (+ 1135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:12:24.051589 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:12:24.051591 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:12:24.051604 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:12:24.051632 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:12:24.051640 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:12:24.051714 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:24.051881 (+ 167us) server_negotiation.cc:299] Negotiation successful
0924 04:12:24.051944 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":84}
I20260924 04:12:24.053016 18288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:24.044532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40763 (local address 127.0.98.2:56629)
0924 04:12:24.045220 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:24.045262 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 04:12:24.046651 (+ 1389us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:12:24.047395 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:12:24.047405 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:12:24.047520 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:12:24.048094 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:24.048108 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:24.048983 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:24.048986 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:24.049976 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:24.049985 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:24.050490 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:24.050493 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:24.050671 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:24.051320 (+ 649us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:12:24.051322 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:12:24.051363 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:12:24.051428 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:12:24.051727 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:12:24.051740 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:12:24.051749 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 04:12:24.051880 (+ 131us) client_negotiation.cc:240] Negotiation successful
0924 04:12:24.052173 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":133,"threads_started":1}
I20260924 04:12:24.054155 18286 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40763
I20260924 04:12:24.054409 18286 heartbeater.cc:464] Registering TS with master...
I20260924 04:12:24.054908 18286 heartbeater.cc:511] Master 127.0.98.62:40763 requested a full tablet report, sending...
I20260924 04:12:24.055956 17967 ts_manager.cc:194] Registered new tserver with Master: ef46fefdd279418a90e7c85cbfa068be (127.0.98.2:44479)
I20260924 04:12:24.056885 17967 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:56629
W20260924 04:12:24.163033 18289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:12:24.163318 18289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:12:24.163358 18289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:12:24.167447 18289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:12:24.167512 18289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:12:24.167570 18289 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:12:24.172582 18289 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222898807720-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222898807720-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222898807720-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40763
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18289
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222898807720-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:12:24.173725 18289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:12:24.175069 18289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:12:24.184640 18295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:12:24.184605 18297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:12:24.184787 18289 server_base.cc:1034] running on GCE node
W20260924 04:12:24.184602 18294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:12:24.185352 18289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:12:24.186030 18289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:12:24.187222 18289 hybrid_clock.cc:648] HybridClock initialized: now 1790223144187209 us; error 31 us; skew 500 ppm
I20260924 04:12:24.189018 18289 webserver.cc:460] Webserver started at http://127.0.98.3:36729/ using document root <none> and password file <none>
I20260924 04:12:24.189617 18289 fs_manager.cc:359] Metadata directory not provided
I20260924 04:12:24.189709 18289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:12:24.189941 18289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:12:24.191911 18289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222898807720-392-0/minicluster-data/ts-2/data/instance:
uuid: "6d59abf6676d4ea4824a5bc0fd64f14b"
format_stamp: "Formatted at 2026-09-24 04:12:24 on dist-test-slave-00ld"
server_key: "0c49a1967ce9efc0c183e76a43be22c9"
server_key_iv: "50997dddde1f0c3e4f6b944a7d4e1485"
server_key_version: "encryptionkey@0"
I20260924 04:12:24.192512 18289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222898807720-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6d59abf6676d4ea4824a5bc0fd64f14b"
format_stamp: "Formatted at 2026-09-24 04:12:24 on dist-test-slave-00ld"
server_key: "0c49a1967ce9efc0c183e76a43be22c9"
server_key_iv: "50997dddde1f0c3e4f6b944a7d4e1485"
server_key_version: "encryptionkey@0"
I20260924 04:12:24.196336 18289 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260924 04:12:24.198871 18303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:12:24.200084 18289 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:12:24.200225 18289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222898807720-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222898807720-392-0/minicluster-data/ts-2/wal
uuid: "6d59abf6676d4ea4824a5bc0fd64f14b"
format_stamp: "Formatted at 2026-09-24 04:12:24 on dist-test-slave-00ld"
server_key: "0c49a1967ce9efc0c183e76a43be22c9"
server_key_iv: "50997dddde1f0c3e4f6b944a7d4e1485"
server_key_version: "encryptionkey@0"
I20260924 04:12:24.200337 18289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222898807720-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222898807720-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222898807720-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:12:24.234666 18289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:12:24.235405 18289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:12:24.235632 18289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:12:24.236284 18289 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:12:24.237488 18289 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:12:24.237576 18289 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:12:24.237653 18289 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:12:24.237689 18289 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:12:24.247135 18289 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46469
I20260924 04:12:24.247154 18416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46469 every 8 connection(s)
I20260924 04:12:24.248102 18289 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222898807720-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:12:24.256237 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 18289
I20260924 04:12:24.256366 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222898807720-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:12:24.256644 392 external_mini_cluster.cc:1428] Setting key 26638bbc56c3c5eaeba9cd40699408e3
I20260924 04:12:24.257315 18006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:24.249983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45485 (local address 127.0.98.62:40763)
0924 04:12:24.250123 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:24.250131 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:24.251115 (+ 984us) server_negotiation.cc:407] Connection header received
0924 04:12:24.252489 (+ 1374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:24.252491 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:24.252539 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:24.252613 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:12:24.253585 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:24.254105 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:24.255268 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:24.255479 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:24.255555 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:24.256815 (+ 1260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:12:24.256819 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:12:24.256823 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:12:24.256838 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:12:24.256860 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:12:24.256871 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:12:24.256945 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:24.257102 (+ 157us) server_negotiation.cc:299] Negotiation successful
0924 04:12:24.257171 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 04:12:24.258363 18419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:24.250332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40763 (local address 127.0.98.3:45485)
0924 04:12:24.250981 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:24.251023 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 04:12:24.252289 (+ 1266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:12:24.252862 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:12:24.252869 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:12:24.252958 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:12:24.253452 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:24.253464 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:24.254239 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:24.254242 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:24.255143 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:24.255152 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:24.255612 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:24.255615 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:24.255763 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:24.256503 (+ 740us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:12:24.256504 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:12:24.256546 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:12:24.256610 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:12:24.256954 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:12:24.256961 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:12:24.256965 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 04:12:24.257068 (+ 103us) client_negotiation.cc:240] Negotiation successful
0924 04:12:24.257351 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":109,"threads_started":1}
I20260924 04:12:24.259464 18417 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40763
I20260924 04:12:24.259670 18417 heartbeater.cc:464] Registering TS with master...
I20260924 04:12:24.260147 18417 heartbeater.cc:511] Master 127.0.98.62:40763 requested a full tablet report, sending...
I20260924 04:12:24.261212 17967 ts_manager.cc:194] Registered new tserver with Master: 6d59abf6676d4ea4824a5bc0fd64f14b (127.0.98.3:46469)
I20260924 04:12:24.261739 17967 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:45485
I20260924 04:12:24.270660 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:12:24.276060 18006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:24.272733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33672 (local address 127.0.98.62:40763)
0924 04:12:24.272888 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:24.272892 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:24.272986 (+ 94us) server_negotiation.cc:407] Connection header received
0924 04:12:24.273089 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:24.273091 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:24.273131 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:24.273188 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:12:24.273467 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:24.273960 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:24.274739 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:24.274940 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:24.275010 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:24.275648 (+ 638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:12:24.275651 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:12:24.275653 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:12:24.275664 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:12:24.275681 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:12:24.275689 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:12:24.275742 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:24.275820 (+ 78us) server_negotiation.cc:299] Negotiation successful
0924 04:12:24.275863 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 04:12:24.278712 17967 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33672:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:12:24.281651 17967 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:12:24.296168 18429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:24.290259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44479 (local address 127.0.0.1:53424)
0924 04:12:24.291468 (+ 1209us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:24.291479 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:12:24.291559 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:12:24.292115 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:12:24.292118 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:12:24.292138 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:12:24.292268 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:24.292275 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:24.293650 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:24.293652 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:24.294680 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:24.294689 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:24.295584 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:24.295588 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:24.295688 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:24.295729 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 04:12:24.295863 (+ 134us) client_negotiation.cc:240] Negotiation successful
0924 04:12:24.296001 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1155,"mutex_wait_us":86,"thread_start_us":77,"threads_started":1}
I20260924 04:12:24.296204 18432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:24.290714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53424 (local address 127.0.98.2:44479)
0924 04:12:24.291237 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:24.291246 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:24.291576 (+ 330us) server_negotiation.cc:407] Connection header received
0924 04:12:24.291651 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:24.291664 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:24.291824 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:24.291981 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:12:24.292398 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:24.293539 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:24.294808 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:24.295454 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:24.295558 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:24.295781 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:24.295890 (+ 109us) server_negotiation.cc:299] Negotiation successful
0924 04:12:24.295989 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":124,"threads_started":1}
I20260924 04:12:24.296700 18428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:24.289927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38537 (local address 127.0.0.1:34808)
0924 04:12:24.290750 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:24.290802 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 04:12:24.290928 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:12:24.291864 (+ 936us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:12:24.291867 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:12:24.291892 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:12:24.292043 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:24.292050 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:24.293590 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:24.293596 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:24.295093 (+ 1497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:24.295103 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:24.296270 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:24.296275 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:24.296394 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:24.296414 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 04:12:24.296477 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 04:12:24.296538 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":646,"mutex_wait_us":94,"thread_start_us":73,"threads_started":1}
I20260924 04:12:24.297187 18433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:24.290715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60530 (local address 127.0.98.3:46469)
0924 04:12:24.291147 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:24.291162 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:24.291226 (+ 64us) server_negotiation.cc:407] Connection header received
0924 04:12:24.291399 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:24.291407 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:24.291641 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:24.291817 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:12:24.292364 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:24.293487 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:24.295222 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:24.296036 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:24.296156 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:24.296343 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:24.296931 (+ 588us) server_negotiation.cc:299] Negotiation successful
0924 04:12:24.297035 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":154,"threads_started":1}
I20260924 04:12:24.296839 18431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:24.289978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46469 (local address 127.0.0.1:60530)
0924 04:12:24.291140 (+ 1162us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:24.291154 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:12:24.291253 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:12:24.291804 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:12:24.291810 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:12:24.291832 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:12:24.292260 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:24.292264 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:24.293614 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:24.293617 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:24.295094 (+ 1477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:24.295103 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:24.296409 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:24.296414 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:24.296499 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:24.296533 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 04:12:24.296606 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 04:12:24.296713 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1108,"mutex_wait_us":176,"thread_start_us":98,"threads_started":1}
I20260924 04:12:24.297329 18430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:24.290051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34808 (local address 127.0.98.1:38537)
0924 04:12:24.291397 (+ 1346us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:24.291405 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:24.291424 (+ 19us) server_negotiation.cc:407] Connection header received
0924 04:12:24.291485 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:24.291489 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:24.291702 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:24.291969 (+ 267us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:12:24.292189 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:24.293451 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:24.295236 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:24.295920 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:24.296009 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:24.296195 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:24.297034 (+ 839us) server_negotiation.cc:299] Negotiation successful
0924 04:12:24.297140 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1269,"thread_start_us":127,"threads_started":1}
I20260924 04:12:24.299865 18220 tablet_service.cc:1467] Processing CreateTablet for tablet 09afb1d9554f43509511743d3c39348f (DEFAULT_TABLE table=test-table [id=484f9b58fed24c93b0eecbcc28651388]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:12:24.299978 18089 tablet_service.cc:1467] Processing CreateTablet for tablet 09afb1d9554f43509511743d3c39348f (DEFAULT_TABLE table=test-table [id=484f9b58fed24c93b0eecbcc28651388]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:12:24.300292 18351 tablet_service.cc:1467] Processing CreateTablet for tablet 09afb1d9554f43509511743d3c39348f (DEFAULT_TABLE table=test-table [id=484f9b58fed24c93b0eecbcc28651388]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:12:24.300994 18220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09afb1d9554f43509511743d3c39348f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:12:24.301026 18089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09afb1d9554f43509511743d3c39348f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:12:24.301147 18351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09afb1d9554f43509511743d3c39348f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:12:24.307797 18434 tablet_bootstrap.cc:492] T 09afb1d9554f43509511743d3c39348f P ef46fefdd279418a90e7c85cbfa068be: Bootstrap starting.
I20260924 04:12:24.309161 18435 tablet_bootstrap.cc:492] T 09afb1d9554f43509511743d3c39348f P b2e26e21d980479caee07e53c16982ed: Bootstrap starting.
I20260924 04:12:24.309981 18434 tablet_bootstrap.cc:654] T 09afb1d9554f43509511743d3c39348f P ef46fefdd279418a90e7c85cbfa068be: Neither blocks nor log segments found. Creating new log.
I20260924 04:12:24.310706 18434 log.cc:824] T 09afb1d9554f43509511743d3c39348f P ef46fefdd279418a90e7c85cbfa068be: Log is configured to *not* fsync() on all Append() calls
I20260924 04:12:24.310926 18435 tablet_bootstrap.cc:654] T 09afb1d9554f43509511743d3c39348f P b2e26e21d980479caee07e53c16982ed: Neither blocks nor log segments found. Creating new log.
I20260924 04:12:24.311803 18435 log.cc:824] T 09afb1d9554f43509511743d3c39348f P b2e26e21d980479caee07e53c16982ed: Log is configured to *not* fsync() on all Append() calls
I20260924 04:12:24.312480 18436 tablet_bootstrap.cc:492] T 09afb1d9554f43509511743d3c39348f P 6d59abf6676d4ea4824a5bc0fd64f14b: Bootstrap starting.
I20260924 04:12:24.312935 18434 tablet_bootstrap.cc:492] T 09afb1d9554f43509511743d3c39348f P ef46fefdd279418a90e7c85cbfa068be: No bootstrap required, opened a new log
I20260924 04:12:24.313140 18434 ts_tablet_manager.cc:1397] T 09afb1d9554f43509511743d3c39348f P ef46fefdd279418a90e7c85cbfa068be: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20260924 04:12:24.313863 18435 tablet_bootstrap.cc:492] T 09afb1d9554f43509511743d3c39348f P b2e26e21d980479caee07e53c16982ed: No bootstrap required, opened a new log
I20260924 04:12:24.314069 18435 ts_tablet_manager.cc:1397] T 09afb1d9554f43509511743d3c39348f P b2e26e21d980479caee07e53c16982ed: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 04:12:24.314594 18436 tablet_bootstrap.cc:654] T 09afb1d9554f43509511743d3c39348f P 6d59abf6676d4ea4824a5bc0fd64f14b: Neither blocks nor log segments found. Creating new log.
I20260924 04:12:24.315418 18436 log.cc:824] T 09afb1d9554f43509511743d3c39348f P 6d59abf6676d4ea4824a5bc0fd64f14b: Log is configured to *not* fsync() on all Append() calls
I20260924 04:12:24.316651 18434 raft_consensus.cc:348] T 09afb1d9554f43509511743d3c39348f P ef46fefdd279418a90e7c85cbfa068be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2e26e21d980479caee07e53c16982ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38537 } } peers { permanent_uuid: "6d59abf6676d4ea4824a5bc0fd64f14b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46469 } } peers { permanent_uuid: "ef46fefdd279418a90e7c85cbfa068be" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44479 } }
I20260924 04:12:24.316948 18434 raft_consensus.cc:374] T 09afb1d9554f43509511743d3c39348f P ef46fefdd279418a90e7c85cbfa068be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:12:24.317024 18434 raft_consensus.cc:729] T 09afb1d9554f43509511743d3c39348f P ef46fefdd279418a90e7c85cbfa068be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef46fefdd279418a90e7c85cbfa068be, State: Initialized, Role: FOLLOWER
I20260924 04:12:24.317420 18436 tablet_bootstrap.cc:492] T 09afb1d9554f43509511743d3c39348f P 6d59abf6676d4ea4824a5bc0fd64f14b: No bootstrap required, opened a new log
I20260924 04:12:24.317596 18436 ts_tablet_manager.cc:1397] T 09afb1d9554f43509511743d3c39348f P 6d59abf6676d4ea4824a5bc0fd64f14b: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 04:12:24.317518 18434 consensus_queue.cc:260] T 09afb1d9554f43509511743d3c39348f P ef46fefdd279418a90e7c85cbfa068be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2e26e21d980479caee07e53c16982ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38537 } } peers { permanent_uuid: "6d59abf6676d4ea4824a5bc0fd64f14b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46469 } } peers { permanent_uuid: "ef46fefdd279418a90e7c85cbfa068be" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44479 } }
I20260924 04:12:24.318167 18286 heartbeater.cc:503] Master 127.0.98.62:40763 was elected leader, sending a full tablet report...
I20260924 04:12:24.318519 18434 ts_tablet_manager.cc:1428] T 09afb1d9554f43509511743d3c39348f P ef46fefdd279418a90e7c85cbfa068be: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260924 04:12:24.318890 18435 raft_consensus.cc:348] T 09afb1d9554f43509511743d3c39348f P b2e26e21d980479caee07e53c16982ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2e26e21d980479caee07e53c16982ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38537 } } peers { permanent_uuid: "6d59abf6676d4ea4824a5bc0fd64f14b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46469 } } peers { permanent_uuid: "ef46fefdd279418a90e7c85cbfa068be" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44479 } }
I20260924 04:12:24.319128 18435 raft_consensus.cc:374] T 09afb1d9554f43509511743d3c39348f P b2e26e21d980479caee07e53c16982ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:12:24.319245 18435 raft_consensus.cc:729] T 09afb1d9554f43509511743d3c39348f P b2e26e21d980479caee07e53c16982ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2e26e21d980479caee07e53c16982ed, State: Initialized, Role: FOLLOWER
I20260924 04:12:24.319687 18435 consensus_queue.cc:260] T 09afb1d9554f43509511743d3c39348f P b2e26e21d980479caee07e53c16982ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2e26e21d980479caee07e53c16982ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38537 } } peers { permanent_uuid: "6d59abf6676d4ea4824a5bc0fd64f14b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46469 } } peers { permanent_uuid: "ef46fefdd279418a90e7c85cbfa068be" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44479 } }
I20260924 04:12:24.320340 18155 heartbeater.cc:503] Master 127.0.98.62:40763 was elected leader, sending a full tablet report...
I20260924 04:12:24.320611 18435 ts_tablet_manager.cc:1428] T 09afb1d9554f43509511743d3c39348f P b2e26e21d980479caee07e53c16982ed: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
I20260924 04:12:24.321688 18436 raft_consensus.cc:348] T 09afb1d9554f43509511743d3c39348f P 6d59abf6676d4ea4824a5bc0fd64f14b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2e26e21d980479caee07e53c16982ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38537 } } peers { permanent_uuid: "6d59abf6676d4ea4824a5bc0fd64f14b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46469 } } peers { permanent_uuid: "ef46fefdd279418a90e7c85cbfa068be" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44479 } }
I20260924 04:12:24.321928 18436 raft_consensus.cc:374] T 09afb1d9554f43509511743d3c39348f P 6d59abf6676d4ea4824a5bc0fd64f14b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:12:24.322031 18436 raft_consensus.cc:729] T 09afb1d9554f43509511743d3c39348f P 6d59abf6676d4ea4824a5bc0fd64f14b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d59abf6676d4ea4824a5bc0fd64f14b, State: Initialized, Role: FOLLOWER
I20260924 04:12:24.322551 18436 consensus_queue.cc:260] T 09afb1d9554f43509511743d3c39348f P 6d59abf6676d4ea4824a5bc0fd64f14b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2e26e21d980479caee07e53c16982ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38537 } } peers { permanent_uuid: "6d59abf6676d4ea4824a5bc0fd64f14b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46469 } } peers { permanent_uuid: "ef46fefdd279418a90e7c85cbfa068be" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44479 } }
I20260924 04:12:24.323215 18417 heartbeater.cc:503] Master 127.0.98.62:40763 was elected leader, sending a full tablet report...
I20260924 04:12:24.323417 18436 ts_tablet_manager.cc:1428] T 09afb1d9554f43509511743d3c39348f P 6d59abf6676d4ea4824a5bc0fd64f14b: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260924 04:12:24.368219 18156 tablet.cc:2369] T 09afb1d9554f43509511743d3c39348f P b2e26e21d980479caee07e53c16982ed: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:12:24.370895 18440 raft_consensus.cc:482] T 09afb1d9554f43509511743d3c39348f P ef46fefdd279418a90e7c85cbfa068be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:12:24.371097 18440 raft_consensus.cc:504] T 09afb1d9554f43509511743d3c39348f P ef46fefdd279418a90e7c85cbfa068be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2e26e21d980479caee07e53c16982ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38537 } } peers { permanent_uuid: "6d59abf6676d4ea4824a5bc0fd64f14b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46469 } } peers { permanent_uuid: "ef46fefdd279418a90e7c85cbfa068be" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44479 } }
I20260924 04:12:24.372313 18440 leader_election.cc:288] T 09afb1d9554f43509511743d3c39348f P ef46fefdd279418a90e7c85cbfa068be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b2e26e21d980479caee07e53c16982ed (127.0.98.1:38537), 6d59abf6676d4ea4824a5bc0fd64f14b (127.0.98.3:46469)
I20260924 04:12:24.376197 18433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:24.372638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40461 (local address 127.0.98.3:46469)
0924 04:12:24.372802 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:24.372809 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:24.372829 (+ 20us) server_negotiation.cc:407] Connection header received
0924 04:12:24.372942 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:24.372945 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:24.372996 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:24.373074 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:12:24.373462 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:24.374093 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:24.375183 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:24.375629 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:24.375707 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:24.375754 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:24.375983 (+ 229us) server_negotiation.cc:299] Negotiation successful
0924 04:12:24.376075 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 04:12:24.376190 18288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:24.372534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46469 (local address 127.0.98.2:40461)
0924 04:12:24.372676 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:24.372711 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 04:12:24.372823 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:12:24.373118 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:12:24.373121 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:12:24.373139 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:12:24.373312 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:24.373320 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:24.374205 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:24.374209 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:24.375060 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:24.375069 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:24.375748 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:24.375752 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:24.375857 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:24.375878 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 04:12:24.375950 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 04:12:24.376021 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55,"mutex_wait_us":5}
I20260924 04:12:24.376312 18430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:24.372817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45555 (local address 127.0.98.1:38537)
0924 04:12:24.372964 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:24.372969 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:24.373204 (+ 235us) server_negotiation.cc:407] Connection header received
0924 04:12:24.373354 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:24.373360 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:24.373422 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:24.373490 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:12:24.373768 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:24.374321 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:24.375342 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:24.375751 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:24.375853 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:24.375909 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:24.376112 (+ 203us) server_negotiation.cc:299] Negotiation successful
0924 04:12:24.376189 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 04:12:24.376345 18443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:24.372534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38537 (local address 127.0.98.2:45555)
0924 04:12:24.373110 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:24.373122 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:12:24.373238 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:12:24.373530 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:12:24.373533 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:12:24.373551 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:12:24.373663 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:24.373669 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:24.374429 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:24.374432 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:24.375227 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:24.375236 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:24.375869 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:24.375871 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:24.375974 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:24.375999 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 04:12:24.376085 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 04:12:24.376147 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"mutex_wait_us":104,"thread_start_us":125,"threads_started":1}
I20260924 04:12:24.376902 18371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09afb1d9554f43509511743d3c39348f" candidate_uuid: "ef46fefdd279418a90e7c85cbfa068be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d59abf6676d4ea4824a5bc0fd64f14b" is_pre_election: true
I20260924 04:12:24.377110 18109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09afb1d9554f43509511743d3c39348f" candidate_uuid: "ef46fefdd279418a90e7c85cbfa068be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2e26e21d980479caee07e53c16982ed" is_pre_election: true
I20260924 04:12:24.377228 18371 raft_consensus.cc:2454] T 09afb1d9554f43509511743d3c39348f P 6d59abf6676d4ea4824a5bc0fd64f14b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef46fefdd279418a90e7c85cbfa068be in term 0.
I20260924 04:12:24.377359 18109 raft_consensus.cc:2454] T 09afb1d9554f43509511743d3c39348f P b2e26e21d980479caee07e53c16982ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef46fefdd279418a90e7c85cbfa068be in term 0.
I20260924 04:12:24.377727 18174 leader_election.cc:302] T 09afb1d9554f43509511743d3c39348f P ef46fefdd279418a90e7c85cbfa068be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d59abf6676d4ea4824a5bc0fd64f14b, ef46fefdd279418a90e7c85cbfa068be; no voters:
I20260924 04:12:24.378001 18440 raft_consensus.cc:2781] T 09afb1d9554f43509511743d3c39348f P ef46fefdd279418a90e7c85cbfa068be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:12:24.378083 18440 raft_consensus.cc:482] T 09afb1d9554f43509511743d3c39348f P ef46fefdd279418a90e7c85cbfa068be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:12:24.378126 18440 raft_consensus.cc:3037] T 09afb1d9554f43509511743d3c39348f P ef46fefdd279418a90e7c85cbfa068be [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:12:24.379117 18440 raft_consensus.cc:504] T 09afb1d9554f43509511743d3c39348f P ef46fefdd279418a90e7c85cbfa068be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2e26e21d980479caee07e53c16982ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38537 } } peers { permanent_uuid: "6d59abf6676d4ea4824a5bc0fd64f14b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46469 } } peers { permanent_uuid: "ef46fefdd279418a90e7c85cbfa068be" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44479 } }
I20260924 04:12:24.379432 18440 leader_election.cc:288] T 09afb1d9554f43509511743d3c39348f P ef46fefdd279418a90e7c85cbfa068be [CANDIDATE]: Term 1 election: Requested vote from peers b2e26e21d980479caee07e53c16982ed (127.0.98.1:38537), 6d59abf6676d4ea4824a5bc0fd64f14b (127.0.98.3:46469)
I20260924 04:12:24.379729 18109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09afb1d9554f43509511743d3c39348f" candidate_uuid: "ef46fefdd279418a90e7c85cbfa068be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2e26e21d980479caee07e53c16982ed"
I20260924 04:12:24.379815 18371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09afb1d9554f43509511743d3c39348f" candidate_uuid: "ef46fefdd279418a90e7c85cbfa068be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d59abf6676d4ea4824a5bc0fd64f14b"
I20260924 04:12:24.379863 18109 raft_consensus.cc:3037] T 09afb1d9554f43509511743d3c39348f P b2e26e21d980479caee07e53c16982ed [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:12:24.379943 18371 raft_consensus.cc:3037] T 09afb1d9554f43509511743d3c39348f P 6d59abf6676d4ea4824a5bc0fd64f14b [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:12:24.381240 18109 raft_consensus.cc:2454] T 09afb1d9554f43509511743d3c39348f P b2e26e21d980479caee07e53c16982ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef46fefdd279418a90e7c85cbfa068be in term 1.
I20260924 04:12:24.381240 18371 raft_consensus.cc:2454] T 09afb1d9554f43509511743d3c39348f P 6d59abf6676d4ea4824a5bc0fd64f14b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef46fefdd279418a90e7c85cbfa068be in term 1.
I20260924 04:12:24.381546 18176 leader_election.cc:302] T 09afb1d9554f43509511743d3c39348f P ef46fefdd279418a90e7c85cbfa068be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b2e26e21d980479caee07e53c16982ed, ef46fefdd279418a90e7c85cbfa068be; no voters:
I20260924 04:12:24.381752 18440 raft_consensus.cc:2781] T 09afb1d9554f43509511743d3c39348f P ef46fefdd279418a90e7c85cbfa068be [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:12:24.382011 18440 raft_consensus.cc:686] T 09afb1d9554f43509511743d3c39348f P ef46fefdd279418a90e7c85cbfa068be [term 1 LEADER]: Becoming Leader. State: Replica: ef46fefdd279418a90e7c85cbfa068be, State: Running, Role: LEADER
I20260924 04:12:24.382349 18440 consensus_queue.cc:237] T 09afb1d9554f43509511743d3c39348f P ef46fefdd279418a90e7c85cbfa068be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2e26e21d980479caee07e53c16982ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38537 } } peers { permanent_uuid: "6d59abf6676d4ea4824a5bc0fd64f14b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46469 } } peers { permanent_uuid: "ef46fefdd279418a90e7c85cbfa068be" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44479 } }
I20260924 04:12:24.385444 17967 catalog_manager.cc:5495] T 09afb1d9554f43509511743d3c39348f P ef46fefdd279418a90e7c85cbfa068be reported cstate change: term changed from 0 to 1, leader changed from <none> to ef46fefdd279418a90e7c85cbfa068be (127.0.98.2). New cstate: current_term: 1 leader_uuid: "ef46fefdd279418a90e7c85cbfa068be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2e26e21d980479caee07e53c16982ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d59abf6676d4ea4824a5bc0fd64f14b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef46fefdd279418a90e7c85cbfa068be" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44479 } health_report { overall_health: HEALTHY } } }
I20260924 04:12:24.417141 18432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:24.413900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53426 (local address 127.0.98.2:44479)
0924 04:12:24.414024 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:24.414028 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:24.414064 (+ 36us) server_negotiation.cc:407] Connection header received
0924 04:12:24.414136 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:24.414137 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:24.414194 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:24.414286 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:12:24.414484 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:24.414970 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:24.415763 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:24.415974 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:24.416064 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:24.416343 (+ 279us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:12:24.416820 (+ 477us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:12:24.416912 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:24.416970 (+ 58us) server_negotiation.cc:299] Negotiation successful
0924 04:12:24.417017 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 04:12:24.422927 18371 raft_consensus.cc:1261] T 09afb1d9554f43509511743d3c39348f P 6d59abf6676d4ea4824a5bc0fd64f14b [term 1 FOLLOWER]: Refusing update from remote peer ef46fefdd279418a90e7c85cbfa068be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:12:24.422928 18109 raft_consensus.cc:1261] T 09afb1d9554f43509511743d3c39348f P b2e26e21d980479caee07e53c16982ed [term 1 FOLLOWER]: Refusing update from remote peer ef46fefdd279418a90e7c85cbfa068be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:12:24.423671 18440 consensus_queue.cc:1035] T 09afb1d9554f43509511743d3c39348f P ef46fefdd279418a90e7c85cbfa068be [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e26e21d980479caee07e53c16982ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:12:24.423875 18444 consensus_queue.cc:1035] T 09afb1d9554f43509511743d3c39348f P ef46fefdd279418a90e7c85cbfa068be [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d59abf6676d4ea4824a5bc0fd64f14b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:12:24.430886 18447 mvcc.cc:205] Tried to move back new op lower bound from 7332753999548989440 to 7332753999392595968. Current Snapshot: MvccSnapshot[applied={T|T < 7332753999548989440}]
I20260924 04:12:24.431907 18449 mvcc.cc:205] Tried to move back new op lower bound from 7332753999548989440 to 7332753999392595968. Current Snapshot: MvccSnapshot[applied={T|T < 7332753999548989440}]
I20260924 04:12:24.439366 17967 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33672:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:12:24.439599 17967 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33672:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:12:24.442167 17967 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fb19d1bcb9274367802d30380e193c47: Sending DeleteTablet for 3 replicas of tablet 09afb1d9554f43509511743d3c39348f
I20260924 04:12:24.442974 18089 tablet_service.cc:1514] Processing DeleteTablet for tablet 09afb1d9554f43509511743d3c39348f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:12:24 UTC) from {username='slave'} at 127.0.0.1:34808
I20260924 04:12:24.443534 18457 tablet_replica.cc:330] T 09afb1d9554f43509511743d3c39348f P b2e26e21d980479caee07e53c16982ed: stopping tablet replica
I20260924 04:12:24.443568 18351 tablet_service.cc:1514] Processing DeleteTablet for tablet 09afb1d9554f43509511743d3c39348f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:12:24 UTC) from {username='slave'} at 127.0.0.1:60530
I20260924 04:12:24.443904 18457 raft_consensus.cc:2229] T 09afb1d9554f43509511743d3c39348f P b2e26e21d980479caee07e53c16982ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:12:24.444146 18458 tablet_replica.cc:330] T 09afb1d9554f43509511743d3c39348f P 6d59abf6676d4ea4824a5bc0fd64f14b: stopping tablet replica
I20260924 04:12:24.444164 18220 tablet_service.cc:1514] Processing DeleteTablet for tablet 09afb1d9554f43509511743d3c39348f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:12:24 UTC) from {username='slave'} at 127.0.0.1:53424
I20260924 04:12:24.444250 18457 raft_consensus.cc:2258] T 09afb1d9554f43509511743d3c39348f P b2e26e21d980479caee07e53c16982ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:12:24.444433 18458 raft_consensus.cc:2229] T 09afb1d9554f43509511743d3c39348f P 6d59abf6676d4ea4824a5bc0fd64f14b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:12:24.444757 18458 raft_consensus.cc:2258] T 09afb1d9554f43509511743d3c39348f P 6d59abf6676d4ea4824a5bc0fd64f14b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:12:24.445358 18459 tablet_replica.cc:330] T 09afb1d9554f43509511743d3c39348f P ef46fefdd279418a90e7c85cbfa068be: stopping tablet replica
I20260924 04:12:24.445399 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 18027
I20260924 04:12:24.446621 18459 raft_consensus.cc:2229] T 09afb1d9554f43509511743d3c39348f P ef46fefdd279418a90e7c85cbfa068be [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:12:24.446982 18459 raft_consensus.cc:2258] T 09afb1d9554f43509511743d3c39348f P ef46fefdd279418a90e7c85cbfa068be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:12:24.447676 18458 ts_tablet_manager.cc:1905] T 09afb1d9554f43509511743d3c39348f P 6d59abf6676d4ea4824a5bc0fd64f14b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:12:24.448412 18459 ts_tablet_manager.cc:1905] T 09afb1d9554f43509511743d3c39348f P ef46fefdd279418a90e7c85cbfa068be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:12:24.450466 18458 ts_tablet_manager.cc:1918] T 09afb1d9554f43509511743d3c39348f P 6d59abf6676d4ea4824a5bc0fd64f14b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:12:24.450596 18458 log.cc:1196] T 09afb1d9554f43509511743d3c39348f P 6d59abf6676d4ea4824a5bc0fd64f14b: Deleting WAL directory at /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222898807720-392-0/minicluster-data/ts-2/wal/wals/09afb1d9554f43509511743d3c39348f
I20260924 04:12:24.450938 18458 ts_tablet_manager.cc:1939] T 09afb1d9554f43509511743d3c39348f P 6d59abf6676d4ea4824a5bc0fd64f14b: Deleting consensus metadata
I20260924 04:12:24.451820 18459 ts_tablet_manager.cc:1918] T 09afb1d9554f43509511743d3c39348f P ef46fefdd279418a90e7c85cbfa068be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:12:24.451952 18459 log.cc:1196] T 09afb1d9554f43509511743d3c39348f P ef46fefdd279418a90e7c85cbfa068be: Deleting WAL directory at /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790222898807720-392-0/minicluster-data/ts-1/wal/wals/09afb1d9554f43509511743d3c39348f
I20260924 04:12:24.452217 17952 catalog_manager.cc:4841] TS 6d59abf6676d4ea4824a5bc0fd64f14b (127.0.98.3:46469): tablet 09afb1d9554f43509511743d3c39348f (table test-table [id=484f9b58fed24c93b0eecbcc28651388]) successfully deleted
I20260924 04:12:24.452304 18459 ts_tablet_manager.cc:1939] T 09afb1d9554f43509511743d3c39348f P ef46fefdd279418a90e7c85cbfa068be: Deleting consensus metadata
I20260924 04:12:24.453400 17953 catalog_manager.cc:4841] TS ef46fefdd279418a90e7c85cbfa068be (127.0.98.2:44479): tablet 09afb1d9554f43509511743d3c39348f (table test-table [id=484f9b58fed24c93b0eecbcc28651388]) successfully deleted
W20260924 04:12:24.454059 17954 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38537 (error 108)
I20260924 04:12:24.454209 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 18158
I20260924 04:12:24.456043 18431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:24.455702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38537 (local address 127.0.0.1:34822)
0924 04:12:24.455860 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:24.455940 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260924 04:12:24.456200 17954 catalog_manager.cc:4584] TS b2e26e21d980479caee07e53c16982ed (127.0.98.1:38537): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 09afb1d9554f43509511743d3c39348f: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38537: connect: Connection refused (error 111)
I20260924 04:12:24.460171 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 18289
I20260924 04:12:24.465668 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 17936
2026-09-24T04:12:24Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1050 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222898807720-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:12:24 dist-test-slave-00ld krb5kdc[18463](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:12:24 dist-test-slave-00ld krb5kdc[18463](info): set up 2 sockets
Sep 24 04:12:24 dist-test-slave-00ld krb5kdc[18463](info): commencing operation
krb5kdc: starting...
W20260924 04:12:25.500520 392 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222898807720-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222898807720-392-0/krb5kdc/test-user.keytab.
Sep 24 04:12:25 dist-test-slave-00ld krb5kdc[18463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:12:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:12:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:12:25.621539 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222898807720-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222898807720-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222898807720-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32775
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32775
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:12:25.730055 18476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:12:25.730356 18476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:12:25.730432 18476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:12:25.734407 18476 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:12:25.734494 18476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:12:25.734534 18476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:12:25.734592 18476 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:12:25.734632 18476 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:12:25.739806 18476 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222898807720-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222898807720-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32775
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222898807720-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:12:25.740907 18476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:12:25.741941 18476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:12:25.750495 18476 server_base.cc:1034] running on GCE node
W20260924 04:12:25.750537 18482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:12:25.750452 18481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:12:25.750736 18484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:12:25.751317 18476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:12:25.752381 18476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:12:25.753564 18476 hybrid_clock.cc:648] HybridClock initialized: now 1790223145753543 us; error 39 us; skew 500 ppm
Sep 24 04:12:25 dist-test-slave-00ld krb5kdc[18463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223145, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:12:25.756336 18476 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:12:25.757261 18476 webserver.cc:460] Webserver started at http://127.0.98.62:41429/ using document root <none> and password file <none>
I20260924 04:12:25.757807 18476 fs_manager.cc:359] Metadata directory not provided
I20260924 04:12:25.757896 18476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:12:25.758126 18476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:12:25.760094 18476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222898807720-392-0/minicluster-data/master-0/data/instance:
uuid: "3d6e34c67e5b4c8fb45ce586729c7e75"
format_stamp: "Formatted at 2026-09-24 04:12:25 on dist-test-slave-00ld"
server_key: "9c9100e3562375d245b585b765aa0dae"
server_key_iv: "6fd822c0379a215bbd24359d21978990"
server_key_version: "encryptionkey@0"
I20260924 04:12:25.760644 18476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222898807720-392-0/minicluster-data/master-0/wal/instance:
uuid: "3d6e34c67e5b4c8fb45ce586729c7e75"
format_stamp: "Formatted at 2026-09-24 04:12:25 on dist-test-slave-00ld"
server_key: "9c9100e3562375d245b585b765aa0dae"
server_key_iv: "6fd822c0379a215bbd24359d21978990"
server_key_version: "encryptionkey@0"
I20260924 04:12:25.764202 18476 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:12:25.766695 18491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:12:25.767850 18476 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:12:25.767983 18476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222898807720-392-0/minicluster-data/master-0/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222898807720-392-0/minicluster-data/master-0/wal
uuid: "3d6e34c67e5b4c8fb45ce586729c7e75"
format_stamp: "Formatted at 2026-09-24 04:12:25 on dist-test-slave-00ld"
server_key: "9c9100e3562375d245b585b765aa0dae"
server_key_iv: "6fd822c0379a215bbd24359d21978990"
server_key_version: "encryptionkey@0"
I20260924 04:12:25.768088 18476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222898807720-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222898807720-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222898807720-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:12:25.790609 18476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:12:25.793932 18476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:12:25.794129 18476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:12:25.801465 18476 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:32775
I20260924 04:12:25.801477 18543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:32775 every 8 connection(s)
I20260924 04:12:25.802501 18476 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:12:25.805423 18544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:12:25.807353 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 18476
I20260924 04:12:25.807512 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222898807720-392-0/minicluster-data/master-0/wal/instance
I20260924 04:12:25.807808 392 external_mini_cluster.cc:1428] Setting key b6bb2ac97c095ff86f9faf9d4f802784
I20260924 04:12:25.811911 18544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d6e34c67e5b4c8fb45ce586729c7e75: Bootstrap starting.
I20260924 04:12:25.814177 18544 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d6e34c67e5b4c8fb45ce586729c7e75: Neither blocks nor log segments found. Creating new log.
I20260924 04:12:25.814906 18544 log.cc:824] T 00000000000000000000000000000000 P 3d6e34c67e5b4c8fb45ce586729c7e75: Log is configured to *not* fsync() on all Append() calls
Sep 24 04:12:25 dist-test-slave-00ld krb5kdc[18463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:12:25.817206 18544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d6e34c67e5b4c8fb45ce586729c7e75: No bootstrap required, opened a new log
I20260924 04:12:25.821494 18544 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d6e34c67e5b4c8fb45ce586729c7e75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d6e34c67e5b4c8fb45ce586729c7e75" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32775 } }
I20260924 04:12:25.821735 18544 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3d6e34c67e5b4c8fb45ce586729c7e75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:12:25.821794 18544 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d6e34c67e5b4c8fb45ce586729c7e75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d6e34c67e5b4c8fb45ce586729c7e75, State: Initialized, Role: FOLLOWER
I20260924 04:12:25.822165 18547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:25.809250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54210 (local address 127.0.98.62:32775)
0924 04:12:25.809696 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:25.809712 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:25.809754 (+ 42us) server_negotiation.cc:407] Connection header received
0924 04:12:25.810381 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:25.810397 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:25.810710 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:25.811056 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:12:25.811862 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:25.812939 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:25.813820 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:25.814363 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:25.814522 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:25.817464 (+ 2942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:12:25.817469 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:12:25.817482 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:12:25.817508 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:12:25.819143 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:12:25.819698 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:12:25.819702 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:12:25.819707 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:12:25.819824 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:12:25.820109 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:12:25.820111 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:12:25.820112 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:12:25.820475 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:12:25.820630 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:25.820956 (+ 326us) server_negotiation.cc:299] Negotiation successful
0924 04:12:25.821256 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":130,"threads_started":1}
I20260924 04:12:25.822207 18544 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d6e34c67e5b4c8fb45ce586729c7e75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d6e34c67e5b4c8fb45ce586729c7e75" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32775 } }
I20260924 04:12:25.822356 18544 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3d6e34c67e5b4c8fb45ce586729c7e75 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:12:25.822418 18544 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d6e34c67e5b4c8fb45ce586729c7e75 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:12:25.822556 18544 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d6e34c67e5b4c8fb45ce586729c7e75 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:12:25.823606 18544 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d6e34c67e5b4c8fb45ce586729c7e75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d6e34c67e5b4c8fb45ce586729c7e75" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32775 } }
I20260924 04:12:25.823946 18544 leader_election.cc:302] T 00000000000000000000000000000000 P 3d6e34c67e5b4c8fb45ce586729c7e75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3d6e34c67e5b4c8fb45ce586729c7e75; no voters:
I20260924 04:12:25.824258 18544 leader_election.cc:288] T 00000000000000000000000000000000 P 3d6e34c67e5b4c8fb45ce586729c7e75 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:12:25.824371 18549 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3d6e34c67e5b4c8fb45ce586729c7e75 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:12:25.824575 18549 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3d6e34c67e5b4c8fb45ce586729c7e75 [term 1 LEADER]: Becoming Leader. State: Replica: 3d6e34c67e5b4c8fb45ce586729c7e75, State: Running, Role: LEADER
I20260924 04:12:25.824947 18549 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d6e34c67e5b4c8fb45ce586729c7e75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d6e34c67e5b4c8fb45ce586729c7e75" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32775 } }
I20260924 04:12:25.825444 18544 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d6e34c67e5b4c8fb45ce586729c7e75 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:12:25.826635 18550 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d6e34c67e5b4c8fb45ce586729c7e75 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3d6e34c67e5b4c8fb45ce586729c7e75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d6e34c67e5b4c8fb45ce586729c7e75" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32775 } } }
I20260924 04:12:25.826635 18551 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d6e34c67e5b4c8fb45ce586729c7e75 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d6e34c67e5b4c8fb45ce586729c7e75. Latest consensus state: current_term: 1 leader_uuid: "3d6e34c67e5b4c8fb45ce586729c7e75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d6e34c67e5b4c8fb45ce586729c7e75" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32775 } } }
I20260924 04:12:25.826785 18550 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d6e34c67e5b4c8fb45ce586729c7e75 [sys.catalog]: This master's current role is: LEADER
I20260924 04:12:25.826785 18551 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d6e34c67e5b4c8fb45ce586729c7e75 [sys.catalog]: This master's current role is: LEADER
W20260924 04:12:25.829955 18564 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3d6e34c67e5b4c8fb45ce586729c7e75: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:12:25.830046 18564 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:12:25.830157 18559 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:12:25.831074 18559 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:12:25.836172 18559 catalog_manager.cc:1347] Generated new cluster ID: 86e7e4ba66cc41718a3ec3b62ba50c8b
I20260924 04:12:25.836257 18559 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:12:25.856150 18559 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:12:25.857475 18559 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:12:25.871511 18559 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3d6e34c67e5b4c8fb45ce586729c7e75: Generated new TSK 0
I20260924 04:12:25.872334 18559 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222898807720-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222898807720-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:12:25.907486 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222898807720-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222898807720-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222898807720-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222898807720-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32775
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222898807720-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:12:26.018584 18570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:12:26.018828 18570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:12:26.018869 18570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:12:26.022863 18570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:12:26.022918 18570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:12:26.023005 18570 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:12:26.028445 18570 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222898807720-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222898807720-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222898807720-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222898807720-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18570
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222898807720-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:12:26.029630 18570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:12:26.030701 18570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:12:26.040282 18576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:12:26.040349 18575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:12:26.040292 18578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:12:26.040704 18570 server_base.cc:1034] running on GCE node
I20260924 04:12:26.041117 18570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:12:26.041831 18570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:12:26.042996 18570 hybrid_clock.cc:648] HybridClock initialized: now 1790223146042981 us; error 29 us; skew 500 ppm
Sep 24 04:12:26 dist-test-slave-00ld krb5kdc[18463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223146, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:12:26.050734 18570 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:12:26.051725 18570 webserver.cc:460] Webserver started at http://127.0.98.1:39541/ using document root <none> and password file <none>
I20260924 04:12:26.052326 18570 fs_manager.cc:359] Metadata directory not provided
I20260924 04:12:26.052390 18570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:12:26.052608 18570 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:12:26.054737 18570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222898807720-392-0/minicluster-data/ts-0/data/instance:
uuid: "76153017c36d4bc19f813d4024051a8e"
format_stamp: "Formatted at 2026-09-24 04:12:26 on dist-test-slave-00ld"
server_key: "609cf632af82becf748d63b20a826e32"
server_key_iv: "c720b2375d08812679e5ecc9cbf2fac0"
server_key_version: "encryptionkey@0"
I20260924 04:12:26.055312 18570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222898807720-392-0/minicluster-data/ts-0/wal/instance:
uuid: "76153017c36d4bc19f813d4024051a8e"
format_stamp: "Formatted at 2026-09-24 04:12:26 on dist-test-slave-00ld"
server_key: "609cf632af82becf748d63b20a826e32"
server_key_iv: "c720b2375d08812679e5ecc9cbf2fac0"
server_key_version: "encryptionkey@0"
I20260924 04:12:26.059044 18570 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 04:12:26.061662 18585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:12:26.063047 18570 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260924 04:12:26.063277 18570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222898807720-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222898807720-392-0/minicluster-data/ts-0/wal
uuid: "76153017c36d4bc19f813d4024051a8e"
format_stamp: "Formatted at 2026-09-24 04:12:26 on dist-test-slave-00ld"
server_key: "609cf632af82becf748d63b20a826e32"
server_key_iv: "c720b2375d08812679e5ecc9cbf2fac0"
server_key_version: "encryptionkey@0"
I20260924 04:12:26.063407 18570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222898807720-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222898807720-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222898807720-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:12:26.073807 18570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:12:26.076731 18570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:12:26.076923 18570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:12:26.077486 18570 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:12:26.078287 18570 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:12:26.078358 18570 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:12:26.078429 18570 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:12:26.078473 18570 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:12:26.087054 18570 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37167
I20260924 04:12:26.087103 18698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37167 every 8 connection(s)
I20260924 04:12:26.087958 18570 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222898807720-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:12:26.094374 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 18570
I20260924 04:12:26.094540 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222898807720-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:12:26.094858 392 external_mini_cluster.cc:1428] Setting key 4ab6dc1885a894e55ea7499820a84418
Sep 24 04:12:26 dist-test-slave-00ld krb5kdc[18463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223146, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:12:26.102056 18547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:26.089797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56333 (local address 127.0.98.62:32775)
0924 04:12:26.089940 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:26.089944 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:26.090730 (+ 786us) server_negotiation.cc:407] Connection header received
0924 04:12:26.091750 (+ 1020us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:26.091754 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:26.091813 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:26.091914 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:12:26.093337 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:26.093872 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:26.095138 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:26.095372 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:26.095482 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:26.098403 (+ 2921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:12:26.098405 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:12:26.098407 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:12:26.098431 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:12:26.099981 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:12:26.100497 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:12:26.100502 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:12:26.100504 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:12:26.100588 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:12:26.100954 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:12:26.100959 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:12:26.100961 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:12:26.101143 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:12:26.101250 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:26.101758 (+ 508us) server_negotiation.cc:299] Negotiation successful
0924 04:12:26.101885 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 04:12:26.102838 18701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:26.090126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32775 (local address 127.0.98.1:56333)
0924 04:12:26.090572 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:26.090628 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 04:12:26.091513 (+ 885us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:12:26.092092 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:12:26.092099 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:12:26.092437 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:12:26.093173 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:26.093196 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:26.094000 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:26.094005 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:26.095018 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:26.095028 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:26.095531 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:26.095535 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:26.095703 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:26.096314 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:12:26.096330 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:12:26.098199 (+ 1869us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:12:26.100101 (+ 1902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:12:26.100107 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:12:26.100123 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:12:26.100392 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:12:26.100702 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:12:26.100709 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:12:26.100711 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:12:26.100857 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:12:26.101258 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:12:26.101267 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:12:26.101510 (+ 243us) client_negotiation.cc:769] Sending connection context
0924 04:12:26.101695 (+ 185us) client_negotiation.cc:240] Negotiation successful
0924 04:12:26.102015 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":111,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 04:12:26.104079 18699 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32775
I20260924 04:12:26.104364 18699 heartbeater.cc:464] Registering TS with master...
I20260924 04:12:26.104950 18699 heartbeater.cc:511] Master 127.0.98.62:32775 requested a full tablet report, sending...
I20260924 04:12:26.106423 18508 ts_manager.cc:194] Registered new tserver with Master: 76153017c36d4bc19f813d4024051a8e (127.0.98.1:37167)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 04:12:26.107676 18508 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:56333
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222898807720-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222898807720-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:12:26.122007 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222898807720-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222898807720-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222898807720-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222898807720-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32775
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222898807720-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:12:26.233125 18704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:12:26.233381 18704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:12:26.233422 18704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:12:26.237478 18704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:12:26.237530 18704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:12:26.237622 18704 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:12:26.242841 18704 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222898807720-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222898807720-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222898807720-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222898807720-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18704
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222898807720-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:12:26.243885 18704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:12:26.244925 18704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:12:26.255133 18704 server_base.cc:1034] running on GCE node
W20260924 04:12:26.255129 18710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:12:26.255101 18712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:12:26.255352 18709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:12:26.255782 18704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:12:26.256521 18704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:12:26.257692 18704 hybrid_clock.cc:648] HybridClock initialized: now 1790223146257669 us; error 35 us; skew 500 ppm
Sep 24 04:12:26 dist-test-slave-00ld krb5kdc[18463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223146, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:12:26.260444 18704 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:12:26.261359 18704 webserver.cc:460] Webserver started at http://127.0.98.2:36361/ using document root <none> and password file <none>
I20260924 04:12:26.261935 18704 fs_manager.cc:359] Metadata directory not provided
I20260924 04:12:26.262027 18704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:12:26.262264 18704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:12:26.264299 18704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222898807720-392-0/minicluster-data/ts-1/data/instance:
uuid: "cece9cb6c90149a085aba06ca189c550"
format_stamp: "Formatted at 2026-09-24 04:12:26 on dist-test-slave-00ld"
server_key: "91d00c3a7771bd3f796899bd98e1796f"
server_key_iv: "29507866d178661e8148d4accb734194"
server_key_version: "encryptionkey@0"
I20260924 04:12:26.264926 18704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222898807720-392-0/minicluster-data/ts-1/wal/instance:
uuid: "cece9cb6c90149a085aba06ca189c550"
format_stamp: "Formatted at 2026-09-24 04:12:26 on dist-test-slave-00ld"
server_key: "91d00c3a7771bd3f796899bd98e1796f"
server_key_iv: "29507866d178661e8148d4accb734194"
server_key_version: "encryptionkey@0"
I20260924 04:12:26.268869 18704 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 04:12:26.271348 18719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:12:26.272635 18704 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:12:26.272784 18704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222898807720-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222898807720-392-0/minicluster-data/ts-1/wal
uuid: "cece9cb6c90149a085aba06ca189c550"
format_stamp: "Formatted at 2026-09-24 04:12:26 on dist-test-slave-00ld"
server_key: "91d00c3a7771bd3f796899bd98e1796f"
server_key_iv: "29507866d178661e8148d4accb734194"
server_key_version: "encryptionkey@0"
I20260924 04:12:26.272897 18704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222898807720-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222898807720-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222898807720-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:12:26.295707 18704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:12:26.298921 18704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:12:26.299132 18704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:12:26.299726 18704 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:12:26.300539 18704 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:12:26.300585 18704 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:12:26.300654 18704 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:12:26.300714 18704 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:12:26.309626 18704 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44825
I20260924 04:12:26.309648 18832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44825 every 8 connection(s)
I20260924 04:12:26.310529 18704 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222898807720-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:12:26.318437 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 18704
I20260924 04:12:26.318610 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222898807720-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:12:26.318898 392 external_mini_cluster.cc:1428] Setting key bbfa26105d5b97155342b397b2cb5345
Sep 24 04:12:26 dist-test-slave-00ld krb5kdc[18463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223146, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:12:26.324097 18547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:26.312356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37163 (local address 127.0.98.62:32775)
0924 04:12:26.312501 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:26.312504 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:26.313361 (+ 857us) server_negotiation.cc:407] Connection header received
0924 04:12:26.314202 (+ 841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:26.314205 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:26.314255 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:26.314330 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:12:26.315458 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:26.315989 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:26.317163 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:26.317378 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:26.317457 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:26.320058 (+ 2601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:12:26.320062 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:12:26.320064 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:12:26.320087 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:12:26.321830 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:12:26.322484 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:12:26.322487 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:12:26.322489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:12:26.322560 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:12:26.322957 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:12:26.322960 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:12:26.322961 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:12:26.323130 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:12:26.323215 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:26.323863 (+ 648us) server_negotiation.cc:299] Negotiation successful
0924 04:12:26.323972 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 04:12:26.325359 18835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:26.312730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32775 (local address 127.0.98.2:37163)
0924 04:12:26.313188 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:26.313233 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 04:12:26.314014 (+ 781us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:12:26.314491 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:12:26.314497 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:12:26.314871 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:12:26.315337 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:26.315348 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:26.316101 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:26.316104 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:26.317059 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:26.317069 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:26.317542 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:26.317546 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:26.317705 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:26.318299 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:12:26.318315 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:12:26.319884 (+ 1569us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:12:26.321961 (+ 2077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:12:26.321970 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:12:26.321988 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:12:26.322368 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:12:26.322655 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:12:26.322660 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:12:26.322665 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:12:26.322859 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:12:26.323240 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:12:26.323256 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:12:26.323579 (+ 323us) client_negotiation.cc:769] Sending connection context
0924 04:12:26.323795 (+ 216us) client_negotiation.cc:240] Negotiation successful
0924 04:12:26.324156 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":115,"threads_started":1}
I20260924 04:12:26.326722 18833 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32775
I20260924 04:12:26.326973 18833 heartbeater.cc:464] Registering TS with master...
I20260924 04:12:26.327472 18833 heartbeater.cc:511] Master 127.0.98.62:32775 requested a full tablet report, sending...
I20260924 04:12:26.328627 18508 ts_manager.cc:194] Registered new tserver with Master: cece9cb6c90149a085aba06ca189c550 (127.0.98.2:44825)
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 04:12:26.329277 18508 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:37163
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222898807720-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222898807720-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:12:26.347184 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222898807720-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222898807720-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222898807720-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222898807720-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32775
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222898807720-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:12:26.456115 18838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:12:26.456391 18838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:12:26.456435 18838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:12:26.460448 18838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:12:26.460515 18838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:12:26.460575 18838 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:12:26.466050 18838 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222898807720-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222898807720-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222898807720-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222898807720-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18838
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222898807720-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:12:26.467163 18838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:12:26.468235 18838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:12:26.478049 18844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:12:26.478039 18843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:12:26.478171 18838 server_base.cc:1034] running on GCE node
W20260924 04:12:26.478039 18846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:12:26.478670 18838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:12:26.479279 18838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:12:26.480429 18838 hybrid_clock.cc:648] HybridClock initialized: now 1790223146480408 us; error 35 us; skew 500 ppm
Sep 24 04:12:26 dist-test-slave-00ld krb5kdc[18463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223146, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:12:26.483095 18838 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:12:26.483990 18838 webserver.cc:460] Webserver started at http://127.0.98.3:38771/ using document root <none> and password file <none>
I20260924 04:12:26.484545 18838 fs_manager.cc:359] Metadata directory not provided
I20260924 04:12:26.484606 18838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:12:26.484846 18838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:12:26.486821 18838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222898807720-392-0/minicluster-data/ts-2/data/instance:
uuid: "7da574a07dea4f669a61c6c569334f4a"
format_stamp: "Formatted at 2026-09-24 04:12:26 on dist-test-slave-00ld"
server_key: "a24cce1b616f479d1cef908a3d0f05d1"
server_key_iv: "60d3e35591b168d7998e718dd21094d6"
server_key_version: "encryptionkey@0"
I20260924 04:12:26.487377 18838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222898807720-392-0/minicluster-data/ts-2/wal/instance:
uuid: "7da574a07dea4f669a61c6c569334f4a"
format_stamp: "Formatted at 2026-09-24 04:12:26 on dist-test-slave-00ld"
server_key: "a24cce1b616f479d1cef908a3d0f05d1"
server_key_iv: "60d3e35591b168d7998e718dd21094d6"
server_key_version: "encryptionkey@0"
I20260924 04:12:26.491199 18838 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 04:12:26.493791 18853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:12:26.495042 18838 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:12:26.495174 18838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222898807720-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222898807720-392-0/minicluster-data/ts-2/wal
uuid: "7da574a07dea4f669a61c6c569334f4a"
format_stamp: "Formatted at 2026-09-24 04:12:26 on dist-test-slave-00ld"
server_key: "a24cce1b616f479d1cef908a3d0f05d1"
server_key_iv: "60d3e35591b168d7998e718dd21094d6"
server_key_version: "encryptionkey@0"
I20260924 04:12:26.495303 18838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222898807720-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222898807720-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222898807720-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:12:26.508272 18838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:12:26.511170 18838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:12:26.511375 18838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:12:26.511955 18838 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:12:26.512805 18838 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:12:26.512876 18838 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:12:26.512944 18838 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:12:26.513000 18838 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:12:26.521588 18838 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38635
I20260924 04:12:26.521610 18966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38635 every 8 connection(s)
I20260924 04:12:26.522492 18838 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222898807720-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:12:26.523432 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 18838
I20260924 04:12:26.523556 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222898807720-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:12:26.523839 392 external_mini_cluster.cc:1428] Setting key 8866e4314b456db736c5baa017252ffb
Sep 24 04:12:26 dist-test-slave-00ld krb5kdc[18463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223146, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:12:26.535492 18547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:26.524472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46963 (local address 127.0.98.62:32775)
0924 04:12:26.524635 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:26.524639 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:26.525635 (+ 996us) server_negotiation.cc:407] Connection header received
0924 04:12:26.526739 (+ 1104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:26.526742 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:26.526799 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:26.526869 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:12:26.528113 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:26.528652 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:26.529811 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:26.530053 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:26.530170 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:26.532563 (+ 2393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:12:26.532566 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:12:26.532568 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:12:26.532591 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:12:26.533834 (+ 1243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:12:26.534282 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:12:26.534284 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:12:26.534285 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:12:26.534352 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:12:26.534626 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:12:26.534628 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:12:26.534629 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:12:26.534785 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:12:26.534856 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:26.535285 (+ 429us) server_negotiation.cc:299] Negotiation successful
0924 04:12:26.535374 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 04:12:26.536310 18969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:26.524975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32775 (local address 127.0.98.3:46963)
0924 04:12:26.525500 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:26.525543 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 04:12:26.526525 (+ 982us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:12:26.527041 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:12:26.527048 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:12:26.527475 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:12:26.527964 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:26.527977 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:26.528812 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:26.528818 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:26.529685 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:26.529695 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:26.530213 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:26.530217 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:26.530381 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:26.530989 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:12:26.531005 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:12:26.532415 (+ 1410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:12:26.533934 (+ 1519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:12:26.533938 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:12:26.533949 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:12:26.534187 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:12:26.534428 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:12:26.534430 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:12:26.534431 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:12:26.534553 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:12:26.534866 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:12:26.534870 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:12:26.535085 (+ 215us) client_negotiation.cc:769] Sending connection context
0924 04:12:26.535235 (+ 150us) client_negotiation.cc:240] Negotiation successful
0924 04:12:26.535468 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":138,"threads_started":1}
I20260924 04:12:26.537532 18967 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32775
I20260924 04:12:26.537774 18967 heartbeater.cc:464] Registering TS with master...
I20260924 04:12:26.538308 18967 heartbeater.cc:511] Master 127.0.98.62:32775 requested a full tablet report, sending...
I20260924 04:12:26.539305 18508 ts_manager.cc:194] Registered new tserver with Master: 7da574a07dea4f669a61c6c569334f4a (127.0.98.3:38635)
I20260924 04:12:26.539855 18508 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:46963
I20260924 04:12:26.549875 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:12:26.559151 18547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:26.551896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54220 (local address 127.0.98.62:32775)
0924 04:12:26.552026 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:26.552030 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:26.552185 (+ 155us) server_negotiation.cc:407] Connection header received
0924 04:12:26.552301 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:26.552304 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:26.552343 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:26.552400 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:12:26.553142 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:26.553641 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:26.554422 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:26.554622 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:26.554693 (+ 71us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:26.556194 (+ 1501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:12:26.556196 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:12:26.556198 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:12:26.556216 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:12:26.557653 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:12:26.558098 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:12:26.558101 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:12:26.558102 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:12:26.558171 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:12:26.558462 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:12:26.558465 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:12:26.558466 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:12:26.558650 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:12:26.558767 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:26.558966 (+ 199us) server_negotiation.cc:299] Negotiation successful
0924 04:12:26.559038 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 04:12:26.562018 18508 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54220:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:12:26.565065 18508 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:12:26.579404 18981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:26.573823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44825 (local address 127.0.0.1:59194)
0924 04:12:26.574869 (+ 1046us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:26.574881 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:12:26.574954 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:12:26.575553 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:12:26.575556 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:12:26.575572 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:12:26.575673 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:26.575700 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:26.576941 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:26.576945 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:26.578194 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:26.578199 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:26.579061 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:26.579064 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:26.579155 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:26.579180 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 04:12:26.579252 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 04:12:26.579302 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":987,"mutex_wait_us":203,"thread_start_us":84,"threads_started":1}
I20260924 04:12:26.579486 18983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:26.574227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59194 (local address 127.0.98.2:44825)
0924 04:12:26.575221 (+ 994us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:26.575228 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:26.575245 (+ 17us) server_negotiation.cc:407] Connection header received
0924 04:12:26.575301 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:26.575304 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:26.575431 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:26.575573 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:12:26.575806 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:26.576760 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:26.578327 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:26.578924 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:26.579058 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:26.579198 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:26.579272 (+ 74us) server_negotiation.cc:299] Negotiation successful
0924 04:12:26.579343 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":912,"thread_start_us":82,"threads_started":1}
I20260924 04:12:26.579406 18978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:26.573075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37167 (local address 127.0.0.1:42086)
0924 04:12:26.573429 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:26.573480 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 04:12:26.573598 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:12:26.574648 (+ 1050us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:12:26.574656 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:12:26.574679 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:12:26.574881 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:26.574890 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:26.576420 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:26.576423 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:26.577897 (+ 1474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:26.577905 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:26.578892 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:26.578895 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:26.578979 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:26.579023 (+ 44us) client_negotiation.cc:769] Sending connection context
0924 04:12:26.579105 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 04:12:26.579249 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"mutex_wait_us":55,"thread_start_us":92,"threads_started":1}
I20260924 04:12:26.580083 18980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:26.573360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38635 (local address 127.0.0.1:57424)
0924 04:12:26.573834 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:26.573848 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:12:26.573930 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:12:26.574794 (+ 864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:12:26.574797 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:12:26.574817 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:12:26.574947 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:26.574953 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:26.576757 (+ 1804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:26.576761 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:26.577756 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:26.577765 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:26.578847 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:26.578851 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:26.578961 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:26.579018 (+ 57us) client_negotiation.cc:769] Sending connection context
0924 04:12:26.579121 (+ 103us) client_negotiation.cc:240] Negotiation successful
0924 04:12:26.579943 (+ 822us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"mutex_wait_us":704,"thread_start_us":104,"threads_started":1}
I20260924 04:12:26.580319 18979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:26.573211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42086 (local address 127.0.98.1:37167)
0924 04:12:26.574155 (+ 944us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:26.574174 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:26.574199 (+ 25us) server_negotiation.cc:407] Connection header received
0924 04:12:26.574280 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:26.574287 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:26.574499 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:26.574622 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:12:26.574986 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:26.576265 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:26.578005 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:26.578755 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:26.579477 (+ 722us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:26.580030 (+ 553us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:26.580087 (+ 57us) server_negotiation.cc:299] Negotiation successful
0924 04:12:26.580172 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":839,"thread_start_us":117,"threads_started":1}
I20260924 04:12:26.581305 18982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:26.573911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57424 (local address 127.0.98.3:38635)
0924 04:12:26.574332 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:26.574344 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:26.574366 (+ 22us) server_negotiation.cc:407] Connection header received
0924 04:12:26.574439 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:26.574445 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:26.574644 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:26.574800 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:12:26.575039 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:26.576265 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:26.577881 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:26.578702 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:26.578835 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:26.578990 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:26.579223 (+ 233us) server_negotiation.cc:299] Negotiation successful
0924 04:12:26.579331 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":111,"threads_started":1}
I20260924 04:12:26.582458 18767 tablet_service.cc:1467] Processing CreateTablet for tablet 5f42754f0126411eb55c18d6b6615752 (DEFAULT_TABLE table=test-table [id=b50ca4ea89464ef3887ff5b35ec12408]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:12:26.583580 18767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f42754f0126411eb55c18d6b6615752. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:12:26.583204 18633 tablet_service.cc:1467] Processing CreateTablet for tablet 5f42754f0126411eb55c18d6b6615752 (DEFAULT_TABLE table=test-table [id=b50ca4ea89464ef3887ff5b35ec12408]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:12:26.584158 18633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f42754f0126411eb55c18d6b6615752. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:12:26.586802 18901 tablet_service.cc:1467] Processing CreateTablet for tablet 5f42754f0126411eb55c18d6b6615752 (DEFAULT_TABLE table=test-table [id=b50ca4ea89464ef3887ff5b35ec12408]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:12:26.587656 18901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f42754f0126411eb55c18d6b6615752. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:12:26.590927 18985 tablet_bootstrap.cc:492] T 5f42754f0126411eb55c18d6b6615752 P cece9cb6c90149a085aba06ca189c550: Bootstrap starting.
I20260924 04:12:26.591328 18984 tablet_bootstrap.cc:492] T 5f42754f0126411eb55c18d6b6615752 P 76153017c36d4bc19f813d4024051a8e: Bootstrap starting.
I20260924 04:12:26.593632 18984 tablet_bootstrap.cc:654] T 5f42754f0126411eb55c18d6b6615752 P 76153017c36d4bc19f813d4024051a8e: Neither blocks nor log segments found. Creating new log.
I20260924 04:12:26.593737 18986 tablet_bootstrap.cc:492] T 5f42754f0126411eb55c18d6b6615752 P 7da574a07dea4f669a61c6c569334f4a: Bootstrap starting.
I20260924 04:12:26.594986 18985 tablet_bootstrap.cc:654] T 5f42754f0126411eb55c18d6b6615752 P cece9cb6c90149a085aba06ca189c550: Neither blocks nor log segments found. Creating new log.
I20260924 04:12:26.595234 18984 log.cc:824] T 5f42754f0126411eb55c18d6b6615752 P 76153017c36d4bc19f813d4024051a8e: Log is configured to *not* fsync() on all Append() calls
I20260924 04:12:26.595762 18985 log.cc:824] T 5f42754f0126411eb55c18d6b6615752 P cece9cb6c90149a085aba06ca189c550: Log is configured to *not* fsync() on all Append() calls
I20260924 04:12:26.595849 18986 tablet_bootstrap.cc:654] T 5f42754f0126411eb55c18d6b6615752 P 7da574a07dea4f669a61c6c569334f4a: Neither blocks nor log segments found. Creating new log.
I20260924 04:12:26.596935 18986 log.cc:824] T 5f42754f0126411eb55c18d6b6615752 P 7da574a07dea4f669a61c6c569334f4a: Log is configured to *not* fsync() on all Append() calls
I20260924 04:12:26.597183 18984 tablet_bootstrap.cc:492] T 5f42754f0126411eb55c18d6b6615752 P 76153017c36d4bc19f813d4024051a8e: No bootstrap required, opened a new log
I20260924 04:12:26.597383 18984 ts_tablet_manager.cc:1397] T 5f42754f0126411eb55c18d6b6615752 P 76153017c36d4bc19f813d4024051a8e: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 04:12:26.597705 18985 tablet_bootstrap.cc:492] T 5f42754f0126411eb55c18d6b6615752 P cece9cb6c90149a085aba06ca189c550: No bootstrap required, opened a new log
I20260924 04:12:26.597898 18985 ts_tablet_manager.cc:1397] T 5f42754f0126411eb55c18d6b6615752 P cece9cb6c90149a085aba06ca189c550: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260924 04:12:26.600910 18984 raft_consensus.cc:348] T 5f42754f0126411eb55c18d6b6615752 P 76153017c36d4bc19f813d4024051a8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76153017c36d4bc19f813d4024051a8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37167 } } peers { permanent_uuid: "cece9cb6c90149a085aba06ca189c550" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44825 } } peers { permanent_uuid: "7da574a07dea4f669a61c6c569334f4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38635 } }
I20260924 04:12:26.601140 18984 raft_consensus.cc:374] T 5f42754f0126411eb55c18d6b6615752 P 76153017c36d4bc19f813d4024051a8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:12:26.601219 18984 raft_consensus.cc:729] T 5f42754f0126411eb55c18d6b6615752 P 76153017c36d4bc19f813d4024051a8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76153017c36d4bc19f813d4024051a8e, State: Initialized, Role: FOLLOWER
I20260924 04:12:26.601094 18985 raft_consensus.cc:348] T 5f42754f0126411eb55c18d6b6615752 P cece9cb6c90149a085aba06ca189c550 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76153017c36d4bc19f813d4024051a8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37167 } } peers { permanent_uuid: "cece9cb6c90149a085aba06ca189c550" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44825 } } peers { permanent_uuid: "7da574a07dea4f669a61c6c569334f4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38635 } }
I20260924 04:12:26.601387 18985 raft_consensus.cc:374] T 5f42754f0126411eb55c18d6b6615752 P cece9cb6c90149a085aba06ca189c550 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:12:26.601491 18985 raft_consensus.cc:729] T 5f42754f0126411eb55c18d6b6615752 P cece9cb6c90149a085aba06ca189c550 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cece9cb6c90149a085aba06ca189c550, State: Initialized, Role: FOLLOWER
I20260924 04:12:26.601698 18984 consensus_queue.cc:260] T 5f42754f0126411eb55c18d6b6615752 P 76153017c36d4bc19f813d4024051a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76153017c36d4bc19f813d4024051a8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37167 } } peers { permanent_uuid: "cece9cb6c90149a085aba06ca189c550" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44825 } } peers { permanent_uuid: "7da574a07dea4f669a61c6c569334f4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38635 } }
I20260924 04:12:26.601954 18985 consensus_queue.cc:260] T 5f42754f0126411eb55c18d6b6615752 P cece9cb6c90149a085aba06ca189c550 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76153017c36d4bc19f813d4024051a8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37167 } } peers { permanent_uuid: "cece9cb6c90149a085aba06ca189c550" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44825 } } peers { permanent_uuid: "7da574a07dea4f669a61c6c569334f4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38635 } }
I20260924 04:12:26.602365 18699 heartbeater.cc:503] Master 127.0.98.62:32775 was elected leader, sending a full tablet report...
I20260924 04:12:26.602583 18984 ts_tablet_manager.cc:1428] T 5f42754f0126411eb55c18d6b6615752 P 76153017c36d4bc19f813d4024051a8e: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260924 04:12:26.602860 18985 ts_tablet_manager.cc:1428] T 5f42754f0126411eb55c18d6b6615752 P cece9cb6c90149a085aba06ca189c550: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260924 04:12:26.603012 18833 heartbeater.cc:503] Master 127.0.98.62:32775 was elected leader, sending a full tablet report...
I20260924 04:12:26.604485 18986 tablet_bootstrap.cc:492] T 5f42754f0126411eb55c18d6b6615752 P 7da574a07dea4f669a61c6c569334f4a: No bootstrap required, opened a new log
I20260924 04:12:26.604732 18986 ts_tablet_manager.cc:1397] T 5f42754f0126411eb55c18d6b6615752 P 7da574a07dea4f669a61c6c569334f4a: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20260924 04:12:26.608012 18986 raft_consensus.cc:348] T 5f42754f0126411eb55c18d6b6615752 P 7da574a07dea4f669a61c6c569334f4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76153017c36d4bc19f813d4024051a8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37167 } } peers { permanent_uuid: "cece9cb6c90149a085aba06ca189c550" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44825 } } peers { permanent_uuid: "7da574a07dea4f669a61c6c569334f4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38635 } }
I20260924 04:12:26.608237 18986 raft_consensus.cc:374] T 5f42754f0126411eb55c18d6b6615752 P 7da574a07dea4f669a61c6c569334f4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:12:26.608300 18986 raft_consensus.cc:729] T 5f42754f0126411eb55c18d6b6615752 P 7da574a07dea4f669a61c6c569334f4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7da574a07dea4f669a61c6c569334f4a, State: Initialized, Role: FOLLOWER
I20260924 04:12:26.608660 18986 consensus_queue.cc:260] T 5f42754f0126411eb55c18d6b6615752 P 7da574a07dea4f669a61c6c569334f4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76153017c36d4bc19f813d4024051a8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37167 } } peers { permanent_uuid: "cece9cb6c90149a085aba06ca189c550" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44825 } } peers { permanent_uuid: "7da574a07dea4f669a61c6c569334f4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38635 } }
I20260924 04:12:26.609256 18967 heartbeater.cc:503] Master 127.0.98.62:32775 was elected leader, sending a full tablet report...
I20260924 04:12:26.609416 18986 ts_tablet_manager.cc:1428] T 5f42754f0126411eb55c18d6b6615752 P 7da574a07dea4f669a61c6c569334f4a: Time spent starting tablet: real 0.005s user 0.001s sys 0.003s
I20260924 04:12:26.619154 18991 raft_consensus.cc:482] T 5f42754f0126411eb55c18d6b6615752 P cece9cb6c90149a085aba06ca189c550 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:12:26.619319 18991 raft_consensus.cc:504] T 5f42754f0126411eb55c18d6b6615752 P cece9cb6c90149a085aba06ca189c550 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76153017c36d4bc19f813d4024051a8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37167 } } peers { permanent_uuid: "cece9cb6c90149a085aba06ca189c550" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44825 } } peers { permanent_uuid: "7da574a07dea4f669a61c6c569334f4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38635 } }
I20260924 04:12:26.620316 18991 leader_election.cc:288] T 5f42754f0126411eb55c18d6b6615752 P cece9cb6c90149a085aba06ca189c550 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76153017c36d4bc19f813d4024051a8e (127.0.98.1:37167), 7da574a07dea4f669a61c6c569334f4a (127.0.98.3:38635)
I20260924 04:12:26.624924 18982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:26.620624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48001 (local address 127.0.98.3:38635)
0924 04:12:26.620773 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:26.620777 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:26.621126 (+ 349us) server_negotiation.cc:407] Connection header received
0924 04:12:26.621228 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:26.621230 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:26.621270 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:26.621366 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:12:26.621591 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:26.622299 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:26.623628 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:26.624244 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:26.624354 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:26.624416 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:26.624731 (+ 315us) server_negotiation.cc:299] Negotiation successful
0924 04:12:26.624796 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 04:12:26.624924 18979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:26.620632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60205 (local address 127.0.98.1:37167)
0924 04:12:26.620769 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:26.620775 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:26.620792 (+ 17us) server_negotiation.cc:407] Connection header received
0924 04:12:26.620898 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:26.620901 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:26.620940 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:26.621011 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:12:26.621349 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:26.622119 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:26.623636 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:26.624244 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:26.624354 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:26.624417 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:26.624724 (+ 307us) server_negotiation.cc:299] Negotiation successful
0924 04:12:26.624796 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 04:12:26.624963 18835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:26.620537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37167 (local address 127.0.98.2:60205)
0924 04:12:26.620649 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:26.620665 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:12:26.620782 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:12:26.621049 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:12:26.621052 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:12:26.621076 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:12:26.621223 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:26.621229 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:26.622223 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:26.622225 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:26.623483 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:26.623495 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:26.624365 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:26.624369 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:26.624533 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:26.624591 (+ 58us) client_negotiation.cc:769] Sending connection context
0924 04:12:26.624670 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 04:12:26.624801 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":64,"spinlock_wait_cycles":15104}
I20260924 04:12:26.624971 18993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:26.620537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38635 (local address 127.0.98.2:48001)
0924 04:12:26.621029 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:26.621041 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:12:26.621139 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:12:26.621384 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:12:26.621386 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:12:26.621402 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:12:26.621491 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:26.621495 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:26.622388 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:26.622391 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:26.623515 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:26.623525 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:26.624365 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:26.624369 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:26.624495 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:26.624597 (+ 102us) client_negotiation.cc:769] Sending connection context
0924 04:12:26.624670 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 04:12:26.624801 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"mutex_wait_us":185,"spinlock_wait_cycles":768,"thread_start_us":161,"threads_started":1}
I20260924 04:12:26.625728 18921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f42754f0126411eb55c18d6b6615752" candidate_uuid: "cece9cb6c90149a085aba06ca189c550" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da574a07dea4f669a61c6c569334f4a" is_pre_election: true
I20260924 04:12:26.625734 18653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f42754f0126411eb55c18d6b6615752" candidate_uuid: "cece9cb6c90149a085aba06ca189c550" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76153017c36d4bc19f813d4024051a8e" is_pre_election: true
I20260924 04:12:26.626040 18653 raft_consensus.cc:2454] T 5f42754f0126411eb55c18d6b6615752 P 76153017c36d4bc19f813d4024051a8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cece9cb6c90149a085aba06ca189c550 in term 0.
I20260924 04:12:26.626040 18921 raft_consensus.cc:2454] T 5f42754f0126411eb55c18d6b6615752 P 7da574a07dea4f669a61c6c569334f4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cece9cb6c90149a085aba06ca189c550 in term 0.
I20260924 04:12:26.626560 18723 leader_election.cc:302] T 5f42754f0126411eb55c18d6b6615752 P cece9cb6c90149a085aba06ca189c550 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76153017c36d4bc19f813d4024051a8e, cece9cb6c90149a085aba06ca189c550; no voters:
I20260924 04:12:26.626860 18991 raft_consensus.cc:2781] T 5f42754f0126411eb55c18d6b6615752 P cece9cb6c90149a085aba06ca189c550 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:12:26.626945 18991 raft_consensus.cc:482] T 5f42754f0126411eb55c18d6b6615752 P cece9cb6c90149a085aba06ca189c550 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:12:26.626986 18991 raft_consensus.cc:3037] T 5f42754f0126411eb55c18d6b6615752 P cece9cb6c90149a085aba06ca189c550 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:12:26.628074 18991 raft_consensus.cc:504] T 5f42754f0126411eb55c18d6b6615752 P cece9cb6c90149a085aba06ca189c550 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76153017c36d4bc19f813d4024051a8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37167 } } peers { permanent_uuid: "cece9cb6c90149a085aba06ca189c550" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44825 } } peers { permanent_uuid: "7da574a07dea4f669a61c6c569334f4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38635 } }
I20260924 04:12:26.628450 18991 leader_election.cc:288] T 5f42754f0126411eb55c18d6b6615752 P cece9cb6c90149a085aba06ca189c550 [CANDIDATE]: Term 1 election: Requested vote from peers 76153017c36d4bc19f813d4024051a8e (127.0.98.1:37167), 7da574a07dea4f669a61c6c569334f4a (127.0.98.3:38635)
I20260924 04:12:26.628861 18921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f42754f0126411eb55c18d6b6615752" candidate_uuid: "cece9cb6c90149a085aba06ca189c550" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da574a07dea4f669a61c6c569334f4a"
I20260924 04:12:26.628854 18653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f42754f0126411eb55c18d6b6615752" candidate_uuid: "cece9cb6c90149a085aba06ca189c550" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76153017c36d4bc19f813d4024051a8e"
I20260924 04:12:26.628985 18653 raft_consensus.cc:3037] T 5f42754f0126411eb55c18d6b6615752 P 76153017c36d4bc19f813d4024051a8e [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:12:26.628984 18921 raft_consensus.cc:3037] T 5f42754f0126411eb55c18d6b6615752 P 7da574a07dea4f669a61c6c569334f4a [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:12:26.630249 18921 raft_consensus.cc:2454] T 5f42754f0126411eb55c18d6b6615752 P 7da574a07dea4f669a61c6c569334f4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cece9cb6c90149a085aba06ca189c550 in term 1.
I20260924 04:12:26.630261 18653 raft_consensus.cc:2454] T 5f42754f0126411eb55c18d6b6615752 P 76153017c36d4bc19f813d4024051a8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cece9cb6c90149a085aba06ca189c550 in term 1.
I20260924 04:12:26.630558 18722 leader_election.cc:302] T 5f42754f0126411eb55c18d6b6615752 P cece9cb6c90149a085aba06ca189c550 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7da574a07dea4f669a61c6c569334f4a, cece9cb6c90149a085aba06ca189c550; no voters:
I20260924 04:12:26.630756 18991 raft_consensus.cc:2781] T 5f42754f0126411eb55c18d6b6615752 P cece9cb6c90149a085aba06ca189c550 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:12:26.630954 18991 raft_consensus.cc:686] T 5f42754f0126411eb55c18d6b6615752 P cece9cb6c90149a085aba06ca189c550 [term 1 LEADER]: Becoming Leader. State: Replica: cece9cb6c90149a085aba06ca189c550, State: Running, Role: LEADER
I20260924 04:12:26.631280 18991 consensus_queue.cc:237] T 5f42754f0126411eb55c18d6b6615752 P cece9cb6c90149a085aba06ca189c550 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76153017c36d4bc19f813d4024051a8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37167 } } peers { permanent_uuid: "cece9cb6c90149a085aba06ca189c550" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44825 } } peers { permanent_uuid: "7da574a07dea4f669a61c6c569334f4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38635 } }
I20260924 04:12:26.634476 18506 catalog_manager.cc:5495] T 5f42754f0126411eb55c18d6b6615752 P cece9cb6c90149a085aba06ca189c550 reported cstate change: term changed from 0 to 1, leader changed from <none> to cece9cb6c90149a085aba06ca189c550 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "cece9cb6c90149a085aba06ca189c550" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76153017c36d4bc19f813d4024051a8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cece9cb6c90149a085aba06ca189c550" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7da574a07dea4f669a61c6c569334f4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38635 } health_report { overall_health: UNKNOWN } } }
I20260924 04:12:26.646670 18983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:26.643364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59204 (local address 127.0.98.2:44825)
0924 04:12:26.643504 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:26.643508 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:26.643520 (+ 12us) server_negotiation.cc:407] Connection header received
0924 04:12:26.643654 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:26.643656 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:26.643721 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:26.643800 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:12:26.644006 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:26.644491 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:26.645346 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:26.645554 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:26.645631 (+ 77us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:26.645881 (+ 250us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:12:26.646263 (+ 382us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:12:26.646372 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:26.646469 (+ 97us) server_negotiation.cc:299] Negotiation successful
0924 04:12:26.646522 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 04:12:26.653263 18921 raft_consensus.cc:1261] T 5f42754f0126411eb55c18d6b6615752 P 7da574a07dea4f669a61c6c569334f4a [term 1 FOLLOWER]: Refusing update from remote peer cece9cb6c90149a085aba06ca189c550: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:12:26.653281 18653 raft_consensus.cc:1261] T 5f42754f0126411eb55c18d6b6615752 P 76153017c36d4bc19f813d4024051a8e [term 1 FOLLOWER]: Refusing update from remote peer cece9cb6c90149a085aba06ca189c550: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:12:26.653936 18994 consensus_queue.cc:1035] T 5f42754f0126411eb55c18d6b6615752 P cece9cb6c90149a085aba06ca189c550 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76153017c36d4bc19f813d4024051a8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:12:26.654099 18991 consensus_queue.cc:1035] T 5f42754f0126411eb55c18d6b6615752 P cece9cb6c90149a085aba06ca189c550 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7da574a07dea4f669a61c6c569334f4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:12:26.660610 18998 mvcc.cc:205] Tried to move back new op lower bound from 7332754008683229184 to 7332754008604438528. Current Snapshot: MvccSnapshot[applied={T|T < 7332754008683229184}]
I20260924 04:12:26.668942 18508 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54220:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:12:26.669088 18508 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54220:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:12:26.671504 18508 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3d6e34c67e5b4c8fb45ce586729c7e75: Sending DeleteTablet for 3 replicas of tablet 5f42754f0126411eb55c18d6b6615752
I20260924 04:12:26.672356 18767 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f42754f0126411eb55c18d6b6615752 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:12:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59194
I20260924 04:12:26.672353 18901 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f42754f0126411eb55c18d6b6615752 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:12:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57424
I20260924 04:12:26.672941 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 18570
I20260924 04:12:26.673784 19007 tablet_replica.cc:330] T 5f42754f0126411eb55c18d6b6615752 P cece9cb6c90149a085aba06ca189c550: stopping tablet replica
I20260924 04:12:26.674173 19007 raft_consensus.cc:2229] T 5f42754f0126411eb55c18d6b6615752 P cece9cb6c90149a085aba06ca189c550 [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:12:26.674517 19007 raft_consensus.cc:2258] T 5f42754f0126411eb55c18d6b6615752 P cece9cb6c90149a085aba06ca189c550 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:12:26.673035 19008 tablet_replica.cc:330] T 5f42754f0126411eb55c18d6b6615752 P 7da574a07dea4f669a61c6c569334f4a: stopping tablet replica
I20260924 04:12:26.675014 19008 raft_consensus.cc:2229] T 5f42754f0126411eb55c18d6b6615752 P 7da574a07dea4f669a61c6c569334f4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:12:26.675312 19008 raft_consensus.cc:2258] T 5f42754f0126411eb55c18d6b6615752 P 7da574a07dea4f669a61c6c569334f4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:12:26.676918 19007 ts_tablet_manager.cc:1905] T 5f42754f0126411eb55c18d6b6615752 P cece9cb6c90149a085aba06ca189c550: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:12:26.676918 19008 ts_tablet_manager.cc:1905] T 5f42754f0126411eb55c18d6b6615752 P 7da574a07dea4f669a61c6c569334f4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:12:26.679664 19007 ts_tablet_manager.cc:1918] T 5f42754f0126411eb55c18d6b6615752 P cece9cb6c90149a085aba06ca189c550: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:12:26.679772 19007 log.cc:1196] T 5f42754f0126411eb55c18d6b6615752 P cece9cb6c90149a085aba06ca189c550: Deleting WAL directory at /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222898807720-392-0/minicluster-data/ts-1/wal/wals/5f42754f0126411eb55c18d6b6615752
I20260924 04:12:26.679682 19008 ts_tablet_manager.cc:1918] T 5f42754f0126411eb55c18d6b6615752 P 7da574a07dea4f669a61c6c569334f4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:12:26.679932 19008 log.cc:1196] T 5f42754f0126411eb55c18d6b6615752 P 7da574a07dea4f669a61c6c569334f4a: Deleting WAL directory at /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790222898807720-392-0/minicluster-data/ts-2/wal/wals/5f42754f0126411eb55c18d6b6615752
I20260924 04:12:26.680143 19007 ts_tablet_manager.cc:1939] T 5f42754f0126411eb55c18d6b6615752 P cece9cb6c90149a085aba06ca189c550: Deleting consensus metadata
I20260924 04:12:26.680406 19008 ts_tablet_manager.cc:1939] T 5f42754f0126411eb55c18d6b6615752 P 7da574a07dea4f669a61c6c569334f4a: Deleting consensus metadata
I20260924 04:12:26.681486 18494 catalog_manager.cc:4841] TS 7da574a07dea4f669a61c6c569334f4a (127.0.98.3:38635): tablet 5f42754f0126411eb55c18d6b6615752 (table test-table [id=b50ca4ea89464ef3887ff5b35ec12408]) successfully deleted
I20260924 04:12:26.681486 18495 catalog_manager.cc:4841] TS cece9cb6c90149a085aba06ca189c550 (127.0.98.2:44825): tablet 5f42754f0126411eb55c18d6b6615752 (table test-table [id=b50ca4ea89464ef3887ff5b35ec12408]) successfully deleted
W20260924 04:12:26.683966 18495 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37167 (error 108)
I20260924 04:12:26.684242 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 18704
I20260924 04:12:26.685787 18980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:26.685404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37167 (local address 127.0.0.1:42098)
0924 04:12:26.685598 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:26.685669 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
W20260924 04:12:26.685962 18495 catalog_manager.cc:4584] TS 76153017c36d4bc19f813d4024051a8e (127.0.98.1:37167): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5f42754f0126411eb55c18d6b6615752: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37167: connect: Connection refused (error 111)
I20260924 04:12:26.690397 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 18838
I20260924 04:12:26.696305 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 18476
2026-09-24T04:12:26Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2231 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-24T04:12:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:12:26Z Disabled control of system clock
I20260924 04:12:26.725749 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222898807720-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222898807720-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222898807720-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39417
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:35679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39417
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 04:12:26.835160 19013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:12:26.835421 19013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:12:26.835458 19013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:12:26.839465 19013 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:12:26.839525 19013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:12:26.839540 19013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:12:26.839556 19013 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:12:26.839619 19013 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:12:26.844409 19013 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222898807720-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222898807720-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39417
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39417
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19013
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222898807720-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:12:26.845589 19013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:12:26.847007 19013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:12:26.855474 19021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:12:26.855525 19019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:12:26.855566 19018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:12:26.855621 19013 server_base.cc:1034] running on GCE node
I20260924 04:12:26.856279 19013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:12:26.857388 19013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:12:26.858548 19013 hybrid_clock.cc:648] HybridClock initialized: now 1790223146858535 us; error 24 us; skew 500 ppm
I20260924 04:12:26.860320 19013 webserver.cc:460] Webserver started at http://127.0.98.62:34657/ using document root <none> and password file <none>
I20260924 04:12:26.860952 19013 fs_manager.cc:359] Metadata directory not provided
I20260924 04:12:26.861055 19013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:12:26.861289 19013 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:12:26.863289 19013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222898807720-392-0/minicluster-data/master-0/data/instance:
uuid: "ecf68793f81e4eeeb289ea74f3d9ed39"
format_stamp: "Formatted at 2026-09-24 04:12:26 on dist-test-slave-00ld"
server_key: "a77713ae3c91467ae2502815fb196a2d"
server_key_iv: "6ecbb086f422cf32d4eddf1058370306"
server_key_version: "encryptionkey@0"
I20260924 04:12:26.863827 19013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222898807720-392-0/minicluster-data/master-0/wal/instance:
uuid: "ecf68793f81e4eeeb289ea74f3d9ed39"
format_stamp: "Formatted at 2026-09-24 04:12:26 on dist-test-slave-00ld"
server_key: "a77713ae3c91467ae2502815fb196a2d"
server_key_iv: "6ecbb086f422cf32d4eddf1058370306"
server_key_version: "encryptionkey@0"
I20260924 04:12:26.867321 19013 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260924 04:12:26.869643 19027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:12:26.870841 19013 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:12:26.870972 19013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222898807720-392-0/minicluster-data/master-0/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222898807720-392-0/minicluster-data/master-0/wal
uuid: "ecf68793f81e4eeeb289ea74f3d9ed39"
format_stamp: "Formatted at 2026-09-24 04:12:26 on dist-test-slave-00ld"
server_key: "a77713ae3c91467ae2502815fb196a2d"
server_key_iv: "6ecbb086f422cf32d4eddf1058370306"
server_key_version: "encryptionkey@0"
I20260924 04:12:26.871074 19013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222898807720-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222898807720-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222898807720-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:12:26.915817 19013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:12:26.916553 19013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:12:26.916782 19013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:12:26.923666 19013 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39417
I20260924 04:12:26.923679 19079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39417 every 8 connection(s)
I20260924 04:12:26.924599 19013 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:12:26.927572 19080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:12:26.931684 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 19013
I20260924 04:12:26.931818 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222898807720-392-0/minicluster-data/master-0/wal/instance
I20260924 04:12:26.932109 392 external_mini_cluster.cc:1428] Setting key 8d5d398416bb6c50c87a023fd1334007
I20260924 04:12:26.933487 19080 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecf68793f81e4eeeb289ea74f3d9ed39: Bootstrap starting.
I20260924 04:12:26.935726 19080 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ecf68793f81e4eeeb289ea74f3d9ed39: Neither blocks nor log segments found. Creating new log.
I20260924 04:12:26.936479 19080 log.cc:824] T 00000000000000000000000000000000 P ecf68793f81e4eeeb289ea74f3d9ed39: Log is configured to *not* fsync() on all Append() calls
I20260924 04:12:26.938256 19080 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecf68793f81e4eeeb289ea74f3d9ed39: No bootstrap required, opened a new log
I20260924 04:12:26.941089 19083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:26.933524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49428 (local address 127.0.98.62:39417)
0924 04:12:26.934174 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:26.934188 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:26.934228 (+ 40us) server_negotiation.cc:407] Connection header received
0924 04:12:26.934844 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:26.934860 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:26.935239 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:26.935507 (+ 268us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:12:26.935898 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:26.936994 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:26.937870 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:26.938375 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:26.938535 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:26.939372 (+ 837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:12:26.939377 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:12:26.939388 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:12:26.939763 (+ 375us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:12:26.939816 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:12:26.939835 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:12:26.939938 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:26.940081 (+ 143us) server_negotiation.cc:299] Negotiation successful
0924 04:12:26.940349 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":131,"threads_started":1}
I20260924 04:12:26.941982 19080 raft_consensus.cc:348] T 00000000000000000000000000000000 P ecf68793f81e4eeeb289ea74f3d9ed39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecf68793f81e4eeeb289ea74f3d9ed39" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39417 } }
I20260924 04:12:26.942179 19080 raft_consensus.cc:374] T 00000000000000000000000000000000 P ecf68793f81e4eeeb289ea74f3d9ed39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:12:26.942243 19080 raft_consensus.cc:729] T 00000000000000000000000000000000 P ecf68793f81e4eeeb289ea74f3d9ed39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecf68793f81e4eeeb289ea74f3d9ed39, State: Initialized, Role: FOLLOWER
I20260924 04:12:26.942682 19080 consensus_queue.cc:260] T 00000000000000000000000000000000 P ecf68793f81e4eeeb289ea74f3d9ed39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecf68793f81e4eeeb289ea74f3d9ed39" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39417 } }
I20260924 04:12:26.942803 19080 raft_consensus.cc:388] T 00000000000000000000000000000000 P ecf68793f81e4eeeb289ea74f3d9ed39 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:12:26.942885 19080 raft_consensus.cc:482] T 00000000000000000000000000000000 P ecf68793f81e4eeeb289ea74f3d9ed39 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:12:26.943002 19080 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ecf68793f81e4eeeb289ea74f3d9ed39 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:12:26.943867 19080 raft_consensus.cc:504] T 00000000000000000000000000000000 P ecf68793f81e4eeeb289ea74f3d9ed39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecf68793f81e4eeeb289ea74f3d9ed39" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39417 } }
I20260924 04:12:26.944134 19080 leader_election.cc:302] T 00000000000000000000000000000000 P ecf68793f81e4eeeb289ea74f3d9ed39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ecf68793f81e4eeeb289ea74f3d9ed39; no voters:
I20260924 04:12:26.944361 19080 leader_election.cc:288] T 00000000000000000000000000000000 P ecf68793f81e4eeeb289ea74f3d9ed39 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:12:26.944478 19085 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ecf68793f81e4eeeb289ea74f3d9ed39 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:12:26.944720 19085 raft_consensus.cc:686] T 00000000000000000000000000000000 P ecf68793f81e4eeeb289ea74f3d9ed39 [term 1 LEADER]: Becoming Leader. State: Replica: ecf68793f81e4eeeb289ea74f3d9ed39, State: Running, Role: LEADER
I20260924 04:12:26.945019 19085 consensus_queue.cc:237] T 00000000000000000000000000000000 P ecf68793f81e4eeeb289ea74f3d9ed39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecf68793f81e4eeeb289ea74f3d9ed39" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39417 } }
I20260924 04:12:26.945571 19080 sys_catalog.cc:564] T 00000000000000000000000000000000 P ecf68793f81e4eeeb289ea74f3d9ed39 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:12:26.946744 19087 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecf68793f81e4eeeb289ea74f3d9ed39 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ecf68793f81e4eeeb289ea74f3d9ed39. Latest consensus state: current_term: 1 leader_uuid: "ecf68793f81e4eeeb289ea74f3d9ed39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecf68793f81e4eeeb289ea74f3d9ed39" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39417 } } }
I20260924 04:12:26.946732 19086 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecf68793f81e4eeeb289ea74f3d9ed39 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ecf68793f81e4eeeb289ea74f3d9ed39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecf68793f81e4eeeb289ea74f3d9ed39" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39417 } } }
I20260924 04:12:26.946885 19086 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecf68793f81e4eeeb289ea74f3d9ed39 [sys.catalog]: This master's current role is: LEADER
I20260924 04:12:26.946885 19087 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecf68793f81e4eeeb289ea74f3d9ed39 [sys.catalog]: This master's current role is: LEADER
I20260924 04:12:26.947359 19094 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:12:26.950573 19094 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:12:26.956063 19094 catalog_manager.cc:1347] Generated new cluster ID: 8d30eefaf8ca47a29d2d7a8368351592
I20260924 04:12:26.956130 19094 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:12:26.976418 19094 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:12:26.977380 19094 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:12:26.987066 19094 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ecf68793f81e4eeeb289ea74f3d9ed39: Generated new TSK 0
I20260924 04:12:26.987890 19094 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:12:27.001093 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222898807720-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222898807720-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222898807720-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222898807720-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39417
--builtin_ntp_servers=127.0.98.20:35679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 04:12:27.124492 19104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:12:27.124816 19104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:12:27.124897 19104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:12:27.128829 19104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:12:27.128901 19104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:12:27.128988 19104 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:12:27.133603 19104 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222898807720-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222898807720-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222898807720-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39417
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19104
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222898807720-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:12:27.134573 19104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:12:27.135916 19104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:12:27.145139 19112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:12:27.145115 19110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:12:27.145129 19109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:12:27.145351 19104 server_base.cc:1034] running on GCE node
I20260924 04:12:27.145797 19104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:12:27.146555 19104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:12:27.147768 19104 hybrid_clock.cc:648] HybridClock initialized: now 1790223147147754 us; error 31 us; skew 500 ppm
I20260924 04:12:27.149556 19104 webserver.cc:460] Webserver started at http://127.0.98.1:45431/ using document root <none> and password file <none>
I20260924 04:12:27.150126 19104 fs_manager.cc:359] Metadata directory not provided
I20260924 04:12:27.150225 19104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:12:27.150454 19104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:12:27.152438 19104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222898807720-392-0/minicluster-data/ts-0/data/instance:
uuid: "ae9a55925f484c29bbb51565adaacc53"
format_stamp: "Formatted at 2026-09-24 04:12:27 on dist-test-slave-00ld"
server_key: "0fe4cb532b8f90ae64866277e9062d5e"
server_key_iv: "4d90d166b7f104c688d35b3eb0899673"
server_key_version: "encryptionkey@0"
I20260924 04:12:27.153019 19104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222898807720-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ae9a55925f484c29bbb51565adaacc53"
format_stamp: "Formatted at 2026-09-24 04:12:27 on dist-test-slave-00ld"
server_key: "0fe4cb532b8f90ae64866277e9062d5e"
server_key_iv: "4d90d166b7f104c688d35b3eb0899673"
server_key_version: "encryptionkey@0"
I20260924 04:12:27.156455 19104 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260924 04:12:27.158746 19118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:12:27.159881 19104 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:12:27.160009 19104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222898807720-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222898807720-392-0/minicluster-data/ts-0/wal
uuid: "ae9a55925f484c29bbb51565adaacc53"
format_stamp: "Formatted at 2026-09-24 04:12:27 on dist-test-slave-00ld"
server_key: "0fe4cb532b8f90ae64866277e9062d5e"
server_key_iv: "4d90d166b7f104c688d35b3eb0899673"
server_key_version: "encryptionkey@0"
I20260924 04:12:27.160115 19104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222898807720-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222898807720-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222898807720-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:12:27.168908 19104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:12:27.169488 19104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:12:27.169669 19104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:12:27.170194 19104 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:12:27.171039 19104 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:12:27.171110 19104 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:12:27.171180 19104 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:12:27.171229 19104 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:12:27.179903 19104 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33299
I20260924 04:12:27.179937 19231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33299 every 8 connection(s)
I20260924 04:12:27.180747 19104 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222898807720-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:12:27.187769 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 19104
I20260924 04:12:27.187913 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222898807720-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:12:27.188270 392 external_mini_cluster.cc:1428] Setting key 25cee17901a5ba844eac485dc32c0774
I20260924 04:12:27.189944 19083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:27.182390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57959 (local address 127.0.98.62:39417)
0924 04:12:27.182519 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:27.182523 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:27.183553 (+ 1030us) server_negotiation.cc:407] Connection header received
0924 04:12:27.184903 (+ 1350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:27.184906 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:27.184950 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:27.185023 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:12:27.185837 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:27.186364 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:27.187584 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:27.187963 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:27.188068 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:27.189453 (+ 1385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:12:27.189457 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:12:27.189460 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:12:27.189472 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:12:27.189493 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:12:27.189509 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:12:27.189587 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:27.189749 (+ 162us) server_negotiation.cc:299] Negotiation successful
0924 04:12:27.189806 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 04:12:27.190157 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222898807720-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222898807720-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222898807720-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222898807720-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39417
--builtin_ntp_servers=127.0.98.20:35679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 04:12:27.190809 19234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:27.182768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39417 (local address 127.0.98.1:57959)
0924 04:12:27.183417 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:27.183455 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 04:12:27.184652 (+ 1197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:12:27.185172 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:12:27.185179 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:12:27.185268 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:12:27.185716 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:27.185728 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:27.186465 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:27.186467 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:27.187435 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:27.187444 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:27.188148 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:27.188154 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:27.188481 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:27.189184 (+ 703us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:12:27.189185 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:12:27.189237 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:12:27.189296 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:12:27.189590 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:12:27.189601 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:12:27.189608 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 04:12:27.189738 (+ 130us) client_negotiation.cc:240] Negotiation successful
0924 04:12:27.189994 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":106,"threads_started":1}
I20260924 04:12:27.192437 19232 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39417
I20260924 04:12:27.192655 19232 heartbeater.cc:464] Registering TS with master...
I20260924 04:12:27.193147 19232 heartbeater.cc:511] Master 127.0.98.62:39417 requested a full tablet report, sending...
I20260924 04:12:27.194550 19037 ts_manager.cc:194] Registered new tserver with Master: ae9a55925f484c29bbb51565adaacc53 (127.0.98.1:33299)
I20260924 04:12:27.195628 19037 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:57959
W20260924 04:12:27.298408 19235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:12:27.298657 19235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:12:27.298748 19235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:12:27.302780 19235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:12:27.302831 19235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:12:27.302884 19235 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:12:27.307626 19235 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222898807720-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222898807720-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222898807720-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39417
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19235
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222898807720-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:12:27.308612 19235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:12:27.310014 19235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:12:27.319136 19240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:12:27.319164 19241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:12:27.319325 19235 server_base.cc:1034] running on GCE node
W20260924 04:12:27.319182 19243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:12:27.319806 19235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:12:27.320451 19235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:12:27.321630 19235 hybrid_clock.cc:648] HybridClock initialized: now 1790223147321616 us; error 25 us; skew 500 ppm
I20260924 04:12:27.323351 19235 webserver.cc:460] Webserver started at http://127.0.98.2:43365/ using document root <none> and password file <none>
I20260924 04:12:27.323925 19235 fs_manager.cc:359] Metadata directory not provided
I20260924 04:12:27.324018 19235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:12:27.324271 19235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:12:27.326820 19235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222898807720-392-0/minicluster-data/ts-1/data/instance:
uuid: "e3951ac2843a4200adb2daf3f828b1d0"
format_stamp: "Formatted at 2026-09-24 04:12:27 on dist-test-slave-00ld"
server_key: "f2b736fca76b6b83cb78517abb69d012"
server_key_iv: "080f1cb5b04bf0a01e56f4ee15bf5e48"
server_key_version: "encryptionkey@0"
I20260924 04:12:27.327538 19235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222898807720-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e3951ac2843a4200adb2daf3f828b1d0"
format_stamp: "Formatted at 2026-09-24 04:12:27 on dist-test-slave-00ld"
server_key: "f2b736fca76b6b83cb78517abb69d012"
server_key_iv: "080f1cb5b04bf0a01e56f4ee15bf5e48"
server_key_version: "encryptionkey@0"
I20260924 04:12:27.332587 19235 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.006s
I20260924 04:12:27.337716 19249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:12:27.339236 19235 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.000s sys 0.003s
I20260924 04:12:27.339376 19235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222898807720-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222898807720-392-0/minicluster-data/ts-1/wal
uuid: "e3951ac2843a4200adb2daf3f828b1d0"
format_stamp: "Formatted at 2026-09-24 04:12:27 on dist-test-slave-00ld"
server_key: "f2b736fca76b6b83cb78517abb69d012"
server_key_iv: "080f1cb5b04bf0a01e56f4ee15bf5e48"
server_key_version: "encryptionkey@0"
I20260924 04:12:27.339515 19235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222898807720-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222898807720-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222898807720-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:12:27.357751 19235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:12:27.358419 19235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:12:27.358565 19235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:12:27.359077 19235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:12:27.359936 19235 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:12:27.359980 19235 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:12:27.360019 19235 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:12:27.360041 19235 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:12:27.368790 19235 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45409
I20260924 04:12:27.368819 19362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45409 every 8 connection(s)
I20260924 04:12:27.369735 19235 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222898807720-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:12:27.376729 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 19235
I20260924 04:12:27.376857 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222898807720-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:12:27.377194 392 external_mini_cluster.cc:1428] Setting key d89d1cd68d4141a9e1527b509143fa38
I20260924 04:12:27.378661 19083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:27.371438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35763 (local address 127.0.98.62:39417)
0924 04:12:27.371578 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:27.371583 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:27.372651 (+ 1068us) server_negotiation.cc:407] Connection header received
0924 04:12:27.374004 (+ 1353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:27.374006 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:27.374050 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:27.374122 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:12:27.374975 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:27.375503 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:27.376600 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:27.376971 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:27.377095 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:27.378168 (+ 1073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:12:27.378172 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:12:27.378175 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:12:27.378187 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:12:27.378216 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:12:27.378225 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:12:27.378302 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:27.378471 (+ 169us) server_negotiation.cc:299] Negotiation successful
0924 04:12:27.378529 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 04:12:27.379099 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222898807720-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222898807720-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222898807720-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222898807720-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39417
--builtin_ntp_servers=127.0.98.20:35679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 04:12:27.379537 19365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:27.371873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39417 (local address 127.0.98.2:35763)
0924 04:12:27.372496 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:27.372535 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 04:12:27.373823 (+ 1288us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:12:27.374252 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:12:27.374258 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:12:27.374339 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:12:27.374859 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:27.374873 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:27.375600 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:27.375602 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:27.376490 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:27.376499 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:27.377117 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:27.377120 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:27.377289 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:27.377913 (+ 624us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:12:27.377915 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:12:27.377952 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:12:27.378011 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:12:27.378315 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:12:27.378326 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:12:27.378337 (+ 11us) client_negotiation.cc:769] Sending connection context
0924 04:12:27.378456 (+ 119us) client_negotiation.cc:240] Negotiation successful
0924 04:12:27.378711 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":116,"threads_started":1}
I20260924 04:12:27.380590 19363 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39417
I20260924 04:12:27.380851 19363 heartbeater.cc:464] Registering TS with master...
I20260924 04:12:27.381331 19363 heartbeater.cc:511] Master 127.0.98.62:39417 requested a full tablet report, sending...
I20260924 04:12:27.382324 19037 ts_manager.cc:194] Registered new tserver with Master: e3951ac2843a4200adb2daf3f828b1d0 (127.0.98.2:45409)
I20260924 04:12:27.383165 19037 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:35763
W20260924 04:12:27.490311 19366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:12:27.490543 19366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:12:27.490579 19366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:12:27.494627 19366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:12:27.494679 19366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:12:27.494733 19366 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:12:27.499364 19366 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222898807720-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222898807720-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222898807720-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39417
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19366
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222898807720-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:12:27.500787 19366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:12:27.502704 19366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:12:27.512923 19374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:12:27.512931 19372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:12:27.513195 19366 server_base.cc:1034] running on GCE node
W20260924 04:12:27.512936 19371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:12:27.513713 19366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:12:27.514393 19366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:12:27.515580 19366 hybrid_clock.cc:648] HybridClock initialized: now 1790223147515567 us; error 28 us; skew 500 ppm
I20260924 04:12:27.517346 19366 webserver.cc:460] Webserver started at http://127.0.98.3:41193/ using document root <none> and password file <none>
I20260924 04:12:27.517905 19366 fs_manager.cc:359] Metadata directory not provided
I20260924 04:12:27.517993 19366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:12:27.518221 19366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:12:27.520201 19366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222898807720-392-0/minicluster-data/ts-2/data/instance:
uuid: "ccbe2ab276d0422eb2346623ae7c4c58"
format_stamp: "Formatted at 2026-09-24 04:12:27 on dist-test-slave-00ld"
server_key: "1eb5439066a05ea2fabb144029bebd3f"
server_key_iv: "dc4bf34ed953a9fb5352860b92c3a038"
server_key_version: "encryptionkey@0"
I20260924 04:12:27.520804 19366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222898807720-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ccbe2ab276d0422eb2346623ae7c4c58"
format_stamp: "Formatted at 2026-09-24 04:12:27 on dist-test-slave-00ld"
server_key: "1eb5439066a05ea2fabb144029bebd3f"
server_key_iv: "dc4bf34ed953a9fb5352860b92c3a038"
server_key_version: "encryptionkey@0"
I20260924 04:12:27.524153 19366 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260924 04:12:27.526505 19380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:12:27.527598 19366 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 04:12:27.527733 19366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222898807720-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222898807720-392-0/minicluster-data/ts-2/wal
uuid: "ccbe2ab276d0422eb2346623ae7c4c58"
format_stamp: "Formatted at 2026-09-24 04:12:27 on dist-test-slave-00ld"
server_key: "1eb5439066a05ea2fabb144029bebd3f"
server_key_iv: "dc4bf34ed953a9fb5352860b92c3a038"
server_key_version: "encryptionkey@0"
I20260924 04:12:27.527838 19366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222898807720-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222898807720-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222898807720-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:12:27.537600 19366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:12:27.538245 19366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:12:27.538434 19366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:12:27.538993 19366 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:12:27.539899 19366 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:12:27.539971 19366 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:12:27.540042 19366 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:12:27.540086 19366 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:12:27.549335 19366 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40737
I20260924 04:12:27.549396 19493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40737 every 8 connection(s)
I20260924 04:12:27.550293 19366 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222898807720-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:12:27.555073 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 19366
I20260924 04:12:27.555163 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222898807720-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:12:27.555445 392 external_mini_cluster.cc:1428] Setting key 349f69ba4c8a7488d0913e6a03949715
I20260924 04:12:27.559760 19083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:27.552058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49847 (local address 127.0.98.62:39417)
0924 04:12:27.552195 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:27.552198 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:27.553263 (+ 1065us) server_negotiation.cc:407] Connection header received
0924 04:12:27.554561 (+ 1298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:27.554564 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:27.554611 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:27.554684 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:12:27.555669 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:27.556387 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:27.557870 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:27.558098 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:27.558230 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:27.559246 (+ 1016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:12:27.559250 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:12:27.559253 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:12:27.559266 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:12:27.559295 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:12:27.559303 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:12:27.559379 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:27.559547 (+ 168us) server_negotiation.cc:299] Negotiation successful
0924 04:12:27.559609 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 04:12:27.560585 19496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:27.552411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39417 (local address 127.0.98.3:49847)
0924 04:12:27.553100 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:27.553140 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 04:12:27.554379 (+ 1239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:12:27.554848 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:12:27.554854 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:12:27.554936 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:12:27.555503 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:27.555525 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:27.556507 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:27.556510 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:27.557753 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:27.557762 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:27.558260 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:27.558264 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:27.558432 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:27.558980 (+ 548us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:12:27.558981 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:12:27.559038 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:12:27.559101 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:12:27.559392 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:12:27.559402 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:12:27.559408 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 04:12:27.559536 (+ 128us) client_negotiation.cc:240] Negotiation successful
0924 04:12:27.559813 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":115,"threads_started":1}
I20260924 04:12:27.561650 19494 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39417
I20260924 04:12:27.561889 19494 heartbeater.cc:464] Registering TS with master...
I20260924 04:12:27.562404 19494 heartbeater.cc:511] Master 127.0.98.62:39417 requested a full tablet report, sending...
I20260924 04:12:27.563475 19037 ts_manager.cc:194] Registered new tserver with Master: ccbe2ab276d0422eb2346623ae7c4c58 (127.0.98.3:40737)
I20260924 04:12:27.564008 19037 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:49847
I20260924 04:12:27.569494 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:12:27.577180 19083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:27.571998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49444 (local address 127.0.98.62:39417)
0924 04:12:27.572125 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:27.572132 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:27.572442 (+ 310us) server_negotiation.cc:407] Connection header received
0924 04:12:27.572578 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:27.572581 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:27.572629 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:27.572722 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:12:27.572997 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:27.573509 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:27.574661 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:27.574888 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:27.574988 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:27.575910 (+ 922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:12:27.575913 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:12:27.575916 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:12:27.575928 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:12:27.575953 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:12:27.575961 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:12:27.576114 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:27.576938 (+ 824us) server_negotiation.cc:299] Negotiation successful
0924 04:12:27.576997 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 04:12:27.580271 19037 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49444:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:12:27.584034 19037 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:12:27.598753 19509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:27.593305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33299 (local address 127.0.0.1:60074)
0924 04:12:27.593718 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:27.593732 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:12:27.593908 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:12:27.594476 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:12:27.594481 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:12:27.594503 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:12:27.594649 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:27.594657 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:27.596101 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:27.596104 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:27.597244 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:27.597251 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:27.598186 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:27.598189 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:27.598311 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:27.598355 (+ 44us) client_negotiation.cc:769] Sending connection context
0924 04:12:27.598474 (+ 119us) client_negotiation.cc:240] Negotiation successful
0924 04:12:27.598586 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"mutex_wait_us":26,"spinlock_wait_cycles":79104,"thread_start_us":106,"threads_started":1}
I20260924 04:12:27.598824 19507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:27.592955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60074 (local address 127.0.98.1:33299)
0924 04:12:27.593288 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:27.593301 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:27.593837 (+ 536us) server_negotiation.cc:407] Connection header received
0924 04:12:27.594061 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:27.594068 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:27.594300 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:27.594476 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:12:27.594780 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:27.595984 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:27.597362 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:27.598070 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:27.598156 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:27.598300 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:27.598560 (+ 260us) server_negotiation.cc:299] Negotiation successful
0924 04:12:27.598663 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":131,"threads_started":1}
I20260924 04:12:27.599447 19508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:27.592673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40737 (local address 127.0.0.1:54116)
0924 04:12:27.593657 (+ 984us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:27.593708 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 04:12:27.593907 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:12:27.595197 (+ 1290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:12:27.595200 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:12:27.595216 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:12:27.595292 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:27.595298 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:27.596626 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:27.596630 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:27.597661 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:27.597671 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:27.599051 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:27.599055 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:27.599143 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:27.599164 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 04:12:27.599255 (+ 91us) client_negotiation.cc:240] Negotiation successful
0924 04:12:27.599321 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":807,"thread_start_us":83,"threads_started":1}
I20260924 04:12:27.600142 19505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:27.593429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45409 (local address 127.0.0.1:48650)
0924 04:12:27.594127 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:27.594141 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:12:27.594231 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:12:27.594669 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:12:27.594672 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:12:27.594689 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:12:27.594788 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:27.594797 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:27.595788 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:27.595795 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:27.597225 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:27.597236 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:27.599772 (+ 2536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:27.599775 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:27.599857 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:27.599880 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 04:12:27.599959 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 04:12:27.600023 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":641,"mutex_wait_us":2,"thread_start_us":91,"threads_started":1}
I20260924 04:12:27.600297 19510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:27.593576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48650 (local address 127.0.98.2:45409)
0924 04:12:27.593952 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:27.593965 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:27.594334 (+ 369us) server_negotiation.cc:407] Connection header received
0924 04:12:27.594415 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:27.594418 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:27.594544 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:27.594692 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:12:27.594886 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:27.595671 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:27.598353 (+ 2682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:27.599170 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:27.599278 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:27.599444 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:27.600008 (+ 564us) server_negotiation.cc:299] Negotiation successful
0924 04:12:27.600107 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":90,"threads_started":1}
I20260924 04:12:27.600353 19506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:27.592789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54116 (local address 127.0.98.3:40737)
0924 04:12:27.594234 (+ 1445us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:27.594244 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:27.594265 (+ 21us) server_negotiation.cc:407] Connection header received
0924 04:12:27.594719 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:27.594727 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:27.594909 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:27.595044 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:12:27.595418 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:27.596512 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:27.597786 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:27.598393 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:27.598495 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:27.598661 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:27.599299 (+ 638us) server_negotiation.cc:299] Negotiation successful
0924 04:12:27.599399 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1342,"thread_start_us":96,"threads_started":1}
I20260924 04:12:27.602046 19166 tablet_service.cc:1467] Processing CreateTablet for tablet 0fb8d89024bc4cb09165e4bebb22f949 (DEFAULT_TABLE table=test-table [id=477bfd94d577425e93bdcc5b771cec38]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:12:27.603034 19166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fb8d89024bc4cb09165e4bebb22f949. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:12:27.603135 19428 tablet_service.cc:1467] Processing CreateTablet for tablet 0fb8d89024bc4cb09165e4bebb22f949 (DEFAULT_TABLE table=test-table [id=477bfd94d577425e93bdcc5b771cec38]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:12:27.604158 19428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fb8d89024bc4cb09165e4bebb22f949. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:12:27.603816 19297 tablet_service.cc:1467] Processing CreateTablet for tablet 0fb8d89024bc4cb09165e4bebb22f949 (DEFAULT_TABLE table=test-table [id=477bfd94d577425e93bdcc5b771cec38]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:12:27.604817 19297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fb8d89024bc4cb09165e4bebb22f949. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:12:27.609230 19511 tablet_bootstrap.cc:492] T 0fb8d89024bc4cb09165e4bebb22f949 P ae9a55925f484c29bbb51565adaacc53: Bootstrap starting.
I20260924 04:12:27.611301 19512 tablet_bootstrap.cc:492] T 0fb8d89024bc4cb09165e4bebb22f949 P ccbe2ab276d0422eb2346623ae7c4c58: Bootstrap starting.
I20260924 04:12:27.611943 19511 tablet_bootstrap.cc:654] T 0fb8d89024bc4cb09165e4bebb22f949 P ae9a55925f484c29bbb51565adaacc53: Neither blocks nor log segments found. Creating new log.
I20260924 04:12:27.612951 19511 log.cc:824] T 0fb8d89024bc4cb09165e4bebb22f949 P ae9a55925f484c29bbb51565adaacc53: Log is configured to *not* fsync() on all Append() calls
I20260924 04:12:27.613003 19513 tablet_bootstrap.cc:492] T 0fb8d89024bc4cb09165e4bebb22f949 P e3951ac2843a4200adb2daf3f828b1d0: Bootstrap starting.
I20260924 04:12:27.614372 19512 tablet_bootstrap.cc:654] T 0fb8d89024bc4cb09165e4bebb22f949 P ccbe2ab276d0422eb2346623ae7c4c58: Neither blocks nor log segments found. Creating new log.
I20260924 04:12:27.614850 19511 tablet_bootstrap.cc:492] T 0fb8d89024bc4cb09165e4bebb22f949 P ae9a55925f484c29bbb51565adaacc53: No bootstrap required, opened a new log
I20260924 04:12:27.615056 19511 ts_tablet_manager.cc:1397] T 0fb8d89024bc4cb09165e4bebb22f949 P ae9a55925f484c29bbb51565adaacc53: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20260924 04:12:27.615175 19513 tablet_bootstrap.cc:654] T 0fb8d89024bc4cb09165e4bebb22f949 P e3951ac2843a4200adb2daf3f828b1d0: Neither blocks nor log segments found. Creating new log.
I20260924 04:12:27.615197 19512 log.cc:824] T 0fb8d89024bc4cb09165e4bebb22f949 P ccbe2ab276d0422eb2346623ae7c4c58: Log is configured to *not* fsync() on all Append() calls
I20260924 04:12:27.615911 19513 log.cc:824] T 0fb8d89024bc4cb09165e4bebb22f949 P e3951ac2843a4200adb2daf3f828b1d0: Log is configured to *not* fsync() on all Append() calls
I20260924 04:12:27.617268 19512 tablet_bootstrap.cc:492] T 0fb8d89024bc4cb09165e4bebb22f949 P ccbe2ab276d0422eb2346623ae7c4c58: No bootstrap required, opened a new log
I20260924 04:12:27.617467 19512 ts_tablet_manager.cc:1397] T 0fb8d89024bc4cb09165e4bebb22f949 P ccbe2ab276d0422eb2346623ae7c4c58: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 04:12:27.617941 19513 tablet_bootstrap.cc:492] T 0fb8d89024bc4cb09165e4bebb22f949 P e3951ac2843a4200adb2daf3f828b1d0: No bootstrap required, opened a new log
I20260924 04:12:27.618119 19513 ts_tablet_manager.cc:1397] T 0fb8d89024bc4cb09165e4bebb22f949 P e3951ac2843a4200adb2daf3f828b1d0: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 04:12:27.619158 19511 raft_consensus.cc:348] T 0fb8d89024bc4cb09165e4bebb22f949 P ae9a55925f484c29bbb51565adaacc53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccbe2ab276d0422eb2346623ae7c4c58" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40737 } } peers { permanent_uuid: "ae9a55925f484c29bbb51565adaacc53" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33299 } } peers { permanent_uuid: "e3951ac2843a4200adb2daf3f828b1d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45409 } }
I20260924 04:12:27.619406 19511 raft_consensus.cc:374] T 0fb8d89024bc4cb09165e4bebb22f949 P ae9a55925f484c29bbb51565adaacc53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:12:27.619458 19511 raft_consensus.cc:729] T 0fb8d89024bc4cb09165e4bebb22f949 P ae9a55925f484c29bbb51565adaacc53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae9a55925f484c29bbb51565adaacc53, State: Initialized, Role: FOLLOWER
I20260924 04:12:27.619895 19511 consensus_queue.cc:260] T 0fb8d89024bc4cb09165e4bebb22f949 P ae9a55925f484c29bbb51565adaacc53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccbe2ab276d0422eb2346623ae7c4c58" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40737 } } peers { permanent_uuid: "ae9a55925f484c29bbb51565adaacc53" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33299 } } peers { permanent_uuid: "e3951ac2843a4200adb2daf3f828b1d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45409 } }
I20260924 04:12:27.620596 19232 heartbeater.cc:503] Master 127.0.98.62:39417 was elected leader, sending a full tablet report...
I20260924 04:12:27.620890 19511 ts_tablet_manager.cc:1428] T 0fb8d89024bc4cb09165e4bebb22f949 P ae9a55925f484c29bbb51565adaacc53: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
I20260924 04:12:27.621729 19512 raft_consensus.cc:348] T 0fb8d89024bc4cb09165e4bebb22f949 P ccbe2ab276d0422eb2346623ae7c4c58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccbe2ab276d0422eb2346623ae7c4c58" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40737 } } peers { permanent_uuid: "ae9a55925f484c29bbb51565adaacc53" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33299 } } peers { permanent_uuid: "e3951ac2843a4200adb2daf3f828b1d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45409 } }
I20260924 04:12:27.621978 19512 raft_consensus.cc:374] T 0fb8d89024bc4cb09165e4bebb22f949 P ccbe2ab276d0422eb2346623ae7c4c58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:12:27.622048 19512 raft_consensus.cc:729] T 0fb8d89024bc4cb09165e4bebb22f949 P ccbe2ab276d0422eb2346623ae7c4c58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccbe2ab276d0422eb2346623ae7c4c58, State: Initialized, Role: FOLLOWER
I20260924 04:12:27.622462 19512 consensus_queue.cc:260] T 0fb8d89024bc4cb09165e4bebb22f949 P ccbe2ab276d0422eb2346623ae7c4c58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccbe2ab276d0422eb2346623ae7c4c58" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40737 } } peers { permanent_uuid: "ae9a55925f484c29bbb51565adaacc53" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33299 } } peers { permanent_uuid: "e3951ac2843a4200adb2daf3f828b1d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45409 } }
I20260924 04:12:27.623116 19494 heartbeater.cc:503] Master 127.0.98.62:39417 was elected leader, sending a full tablet report...
I20260924 04:12:27.623387 19512 ts_tablet_manager.cc:1428] T 0fb8d89024bc4cb09165e4bebb22f949 P ccbe2ab276d0422eb2346623ae7c4c58: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 04:12:27.624001 19513 raft_consensus.cc:348] T 0fb8d89024bc4cb09165e4bebb22f949 P e3951ac2843a4200adb2daf3f828b1d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccbe2ab276d0422eb2346623ae7c4c58" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40737 } } peers { permanent_uuid: "ae9a55925f484c29bbb51565adaacc53" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33299 } } peers { permanent_uuid: "e3951ac2843a4200adb2daf3f828b1d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45409 } }
I20260924 04:12:27.624253 19513 raft_consensus.cc:374] T 0fb8d89024bc4cb09165e4bebb22f949 P e3951ac2843a4200adb2daf3f828b1d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:12:27.624331 19513 raft_consensus.cc:729] T 0fb8d89024bc4cb09165e4bebb22f949 P e3951ac2843a4200adb2daf3f828b1d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3951ac2843a4200adb2daf3f828b1d0, State: Initialized, Role: FOLLOWER
I20260924 04:12:27.624771 19513 consensus_queue.cc:260] T 0fb8d89024bc4cb09165e4bebb22f949 P e3951ac2843a4200adb2daf3f828b1d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccbe2ab276d0422eb2346623ae7c4c58" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40737 } } peers { permanent_uuid: "ae9a55925f484c29bbb51565adaacc53" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33299 } } peers { permanent_uuid: "e3951ac2843a4200adb2daf3f828b1d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45409 } }
I20260924 04:12:27.625699 19513 ts_tablet_manager.cc:1428] T 0fb8d89024bc4cb09165e4bebb22f949 P e3951ac2843a4200adb2daf3f828b1d0: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260924 04:12:27.626036 19363 heartbeater.cc:503] Master 127.0.98.62:39417 was elected leader, sending a full tablet report...
W20260924 04:12:27.681821 19233 tablet.cc:2369] T 0fb8d89024bc4cb09165e4bebb22f949 P ae9a55925f484c29bbb51565adaacc53: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:12:27.777555 19519 raft_consensus.cc:482] T 0fb8d89024bc4cb09165e4bebb22f949 P e3951ac2843a4200adb2daf3f828b1d0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:12:27.777819 19519 raft_consensus.cc:504] T 0fb8d89024bc4cb09165e4bebb22f949 P e3951ac2843a4200adb2daf3f828b1d0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccbe2ab276d0422eb2346623ae7c4c58" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40737 } } peers { permanent_uuid: "ae9a55925f484c29bbb51565adaacc53" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33299 } } peers { permanent_uuid: "e3951ac2843a4200adb2daf3f828b1d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45409 } }
I20260924 04:12:27.779095 19519 leader_election.cc:288] T 0fb8d89024bc4cb09165e4bebb22f949 P e3951ac2843a4200adb2daf3f828b1d0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ccbe2ab276d0422eb2346623ae7c4c58 (127.0.98.3:40737), ae9a55925f484c29bbb51565adaacc53 (127.0.98.1:33299)
I20260924 04:12:27.783990 19506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:27.779514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37923 (local address 127.0.98.3:40737)
0924 04:12:27.779663 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:27.779670 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:27.779976 (+ 306us) server_negotiation.cc:407] Connection header received
0924 04:12:27.780080 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:27.780083 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:27.780119 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:27.780180 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:12:27.780443 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:27.781181 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:27.782908 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:27.783447 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:27.783547 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:27.783617 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:27.783793 (+ 176us) server_negotiation.cc:299] Negotiation successful
0924 04:12:27.783859 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 04:12:27.784024 19507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:27.779494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54597 (local address 127.0.98.1:33299)
0924 04:12:27.779617 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:27.779621 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:27.779638 (+ 17us) server_negotiation.cc:407] Connection header received
0924 04:12:27.779776 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:27.779780 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:27.779838 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:27.779905 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:12:27.780272 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:27.781091 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:27.782909 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:27.783474 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:27.783590 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:27.783647 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:27.783835 (+ 188us) server_negotiation.cc:299] Negotiation successful
0924 04:12:27.783904 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 04:12:27.784024 19520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:27.779377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40737 (local address 127.0.98.2:37923)
0924 04:12:27.779849 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:27.779860 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:12:27.779990 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:12:27.780212 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:12:27.780217 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:12:27.780237 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:12:27.780343 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:27.780348 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:27.781274 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:27.781277 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:27.782757 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:27.782764 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:27.783550 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:27.783554 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:27.783655 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:27.783684 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 04:12:27.783768 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 04:12:27.783839 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"mutex_wait_us":243,"spinlock_wait_cycles":23680,"thread_start_us":126,"threads_started":1}
I20260924 04:12:27.784026 19365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:27.779374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33299 (local address 127.0.98.2:54597)
0924 04:12:27.779504 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:27.779520 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:12:27.779636 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:12:27.779949 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:12:27.779951 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:12:27.779988 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:12:27.780127 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:27.780132 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:27.781213 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:27.781216 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:27.782753 (+ 1537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:27.782764 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:27.783595 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:27.783599 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:27.783713 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:27.783731 (+ 18us) client_negotiation.cc:769] Sending connection context
0924 04:12:27.783818 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 04:12:27.783889 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":144,"spinlock_wait_cycles":5120}
I20260924 04:12:27.784871 19448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fb8d89024bc4cb09165e4bebb22f949" candidate_uuid: "e3951ac2843a4200adb2daf3f828b1d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccbe2ab276d0422eb2346623ae7c4c58" is_pre_election: true
I20260924 04:12:27.784897 19186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fb8d89024bc4cb09165e4bebb22f949" candidate_uuid: "e3951ac2843a4200adb2daf3f828b1d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9a55925f484c29bbb51565adaacc53" is_pre_election: true
I20260924 04:12:27.785179 19186 raft_consensus.cc:2454] T 0fb8d89024bc4cb09165e4bebb22f949 P ae9a55925f484c29bbb51565adaacc53 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3951ac2843a4200adb2daf3f828b1d0 in term 0.
I20260924 04:12:27.785183 19448 raft_consensus.cc:2454] T 0fb8d89024bc4cb09165e4bebb22f949 P ccbe2ab276d0422eb2346623ae7c4c58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3951ac2843a4200adb2daf3f828b1d0 in term 0.
I20260924 04:12:27.785663 19253 leader_election.cc:302] T 0fb8d89024bc4cb09165e4bebb22f949 P e3951ac2843a4200adb2daf3f828b1d0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9a55925f484c29bbb51565adaacc53, e3951ac2843a4200adb2daf3f828b1d0; no voters:
I20260924 04:12:27.785919 19519 raft_consensus.cc:2781] T 0fb8d89024bc4cb09165e4bebb22f949 P e3951ac2843a4200adb2daf3f828b1d0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:12:27.786001 19519 raft_consensus.cc:482] T 0fb8d89024bc4cb09165e4bebb22f949 P e3951ac2843a4200adb2daf3f828b1d0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:12:27.786044 19519 raft_consensus.cc:3037] T 0fb8d89024bc4cb09165e4bebb22f949 P e3951ac2843a4200adb2daf3f828b1d0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:12:27.787021 19519 raft_consensus.cc:504] T 0fb8d89024bc4cb09165e4bebb22f949 P e3951ac2843a4200adb2daf3f828b1d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccbe2ab276d0422eb2346623ae7c4c58" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40737 } } peers { permanent_uuid: "ae9a55925f484c29bbb51565adaacc53" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33299 } } peers { permanent_uuid: "e3951ac2843a4200adb2daf3f828b1d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45409 } }
I20260924 04:12:27.787331 19519 leader_election.cc:288] T 0fb8d89024bc4cb09165e4bebb22f949 P e3951ac2843a4200adb2daf3f828b1d0 [CANDIDATE]: Term 1 election: Requested vote from peers ccbe2ab276d0422eb2346623ae7c4c58 (127.0.98.3:40737), ae9a55925f484c29bbb51565adaacc53 (127.0.98.1:33299)
I20260924 04:12:27.787629 19448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fb8d89024bc4cb09165e4bebb22f949" candidate_uuid: "e3951ac2843a4200adb2daf3f828b1d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccbe2ab276d0422eb2346623ae7c4c58"
I20260924 04:12:27.787648 19186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fb8d89024bc4cb09165e4bebb22f949" candidate_uuid: "e3951ac2843a4200adb2daf3f828b1d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9a55925f484c29bbb51565adaacc53"
I20260924 04:12:27.787750 19448 raft_consensus.cc:3037] T 0fb8d89024bc4cb09165e4bebb22f949 P ccbe2ab276d0422eb2346623ae7c4c58 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:12:27.787763 19186 raft_consensus.cc:3037] T 0fb8d89024bc4cb09165e4bebb22f949 P ae9a55925f484c29bbb51565adaacc53 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:12:27.789008 19186 raft_consensus.cc:2454] T 0fb8d89024bc4cb09165e4bebb22f949 P ae9a55925f484c29bbb51565adaacc53 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3951ac2843a4200adb2daf3f828b1d0 in term 1.
I20260924 04:12:27.789008 19448 raft_consensus.cc:2454] T 0fb8d89024bc4cb09165e4bebb22f949 P ccbe2ab276d0422eb2346623ae7c4c58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3951ac2843a4200adb2daf3f828b1d0 in term 1.
I20260924 04:12:27.789294 19253 leader_election.cc:302] T 0fb8d89024bc4cb09165e4bebb22f949 P e3951ac2843a4200adb2daf3f828b1d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9a55925f484c29bbb51565adaacc53, e3951ac2843a4200adb2daf3f828b1d0; no voters:
I20260924 04:12:27.789451 19519 raft_consensus.cc:2781] T 0fb8d89024bc4cb09165e4bebb22f949 P e3951ac2843a4200adb2daf3f828b1d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:12:27.789678 19519 raft_consensus.cc:686] T 0fb8d89024bc4cb09165e4bebb22f949 P e3951ac2843a4200adb2daf3f828b1d0 [term 1 LEADER]: Becoming Leader. State: Replica: e3951ac2843a4200adb2daf3f828b1d0, State: Running, Role: LEADER
I20260924 04:12:27.790009 19519 consensus_queue.cc:237] T 0fb8d89024bc4cb09165e4bebb22f949 P e3951ac2843a4200adb2daf3f828b1d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccbe2ab276d0422eb2346623ae7c4c58" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40737 } } peers { permanent_uuid: "ae9a55925f484c29bbb51565adaacc53" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33299 } } peers { permanent_uuid: "e3951ac2843a4200adb2daf3f828b1d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45409 } }
I20260924 04:12:27.793282 19037 catalog_manager.cc:5495] T 0fb8d89024bc4cb09165e4bebb22f949 P e3951ac2843a4200adb2daf3f828b1d0 reported cstate change: term changed from 0 to 1, leader changed from <none> to e3951ac2843a4200adb2daf3f828b1d0 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "e3951ac2843a4200adb2daf3f828b1d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccbe2ab276d0422eb2346623ae7c4c58" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9a55925f484c29bbb51565adaacc53" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3951ac2843a4200adb2daf3f828b1d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45409 } health_report { overall_health: HEALTHY } } }
W20260924 04:12:27.801244 19495 tablet.cc:2369] T 0fb8d89024bc4cb09165e4bebb22f949 P ccbe2ab276d0422eb2346623ae7c4c58: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:12:27.837682 19510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:27.834111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48654 (local address 127.0.98.2:45409)
0924 04:12:27.834257 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:27.834261 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:27.834275 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:12:27.834411 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:27.834413 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:27.834468 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:27.834554 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:12:27.834802 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:27.835296 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:27.836166 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:27.836374 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:27.836447 (+ 73us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:27.836728 (+ 281us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:12:27.837273 (+ 545us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:12:27.837392 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:27.837539 (+ 147us) server_negotiation.cc:299] Negotiation successful
0924 04:12:27.837572 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 04:12:27.843592 19448 raft_consensus.cc:1261] T 0fb8d89024bc4cb09165e4bebb22f949 P ccbe2ab276d0422eb2346623ae7c4c58 [term 1 FOLLOWER]: Refusing update from remote peer e3951ac2843a4200adb2daf3f828b1d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:12:27.843612 19186 raft_consensus.cc:1261] T 0fb8d89024bc4cb09165e4bebb22f949 P ae9a55925f484c29bbb51565adaacc53 [term 1 FOLLOWER]: Refusing update from remote peer e3951ac2843a4200adb2daf3f828b1d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:12:27.844312 19521 consensus_queue.cc:1035] T 0fb8d89024bc4cb09165e4bebb22f949 P e3951ac2843a4200adb2daf3f828b1d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9a55925f484c29bbb51565adaacc53" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:12:27.844542 19519 consensus_queue.cc:1035] T 0fb8d89024bc4cb09165e4bebb22f949 P e3951ac2843a4200adb2daf3f828b1d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccbe2ab276d0422eb2346623ae7c4c58" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:12:27.852766 19525 mvcc.cc:205] Tried to move back new op lower bound from 7332754013560107008 to 7332754013350739968. Current Snapshot: MvccSnapshot[applied={T|T < 7332754013560107008}]
I20260924 04:12:27.861356 19037 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49444:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:12:27.861534 19037 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49444:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:12:27.863962 19037 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ecf68793f81e4eeeb289ea74f3d9ed39: Sending DeleteTablet for 3 replicas of tablet 0fb8d89024bc4cb09165e4bebb22f949
I20260924 04:12:27.864784 19428 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fb8d89024bc4cb09165e4bebb22f949 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:12:27 UTC) from {username='slave'} at 127.0.0.1:54116
I20260924 04:12:27.864784 19297 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fb8d89024bc4cb09165e4bebb22f949 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:12:27 UTC) from {username='slave'} at 127.0.0.1:48650
I20260924 04:12:27.864792 19166 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fb8d89024bc4cb09165e4bebb22f949 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:12:27 UTC) from {username='slave'} at 127.0.0.1:60074
I20260924 04:12:27.865285 19534 tablet_replica.cc:330] T 0fb8d89024bc4cb09165e4bebb22f949 P e3951ac2843a4200adb2daf3f828b1d0: stopping tablet replica
I20260924 04:12:27.865540 19535 tablet_replica.cc:330] T 0fb8d89024bc4cb09165e4bebb22f949 P ccbe2ab276d0422eb2346623ae7c4c58: stopping tablet replica
I20260924 04:12:27.865602 19534 raft_consensus.cc:2229] T 0fb8d89024bc4cb09165e4bebb22f949 P e3951ac2843a4200adb2daf3f828b1d0 [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:12:27.865939 19534 raft_consensus.cc:2258] T 0fb8d89024bc4cb09165e4bebb22f949 P e3951ac2843a4200adb2daf3f828b1d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:12:27.866186 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 19104
I20260924 04:12:27.866295 19535 raft_consensus.cc:2229] T 0fb8d89024bc4cb09165e4bebb22f949 P ccbe2ab276d0422eb2346623ae7c4c58 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:12:27.867316 19535 raft_consensus.cc:2258] T 0fb8d89024bc4cb09165e4bebb22f949 P ccbe2ab276d0422eb2346623ae7c4c58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:12:27.869227 19535 ts_tablet_manager.cc:1905] T 0fb8d89024bc4cb09165e4bebb22f949 P ccbe2ab276d0422eb2346623ae7c4c58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:12:27.871663 19535 ts_tablet_manager.cc:1918] T 0fb8d89024bc4cb09165e4bebb22f949 P ccbe2ab276d0422eb2346623ae7c4c58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:12:27.871773 19535 log.cc:1196] T 0fb8d89024bc4cb09165e4bebb22f949 P ccbe2ab276d0422eb2346623ae7c4c58: Deleting WAL directory at /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790222898807720-392-0/minicluster-data/ts-2/wal/wals/0fb8d89024bc4cb09165e4bebb22f949
I20260924 04:12:27.872138 19535 ts_tablet_manager.cc:1939] T 0fb8d89024bc4cb09165e4bebb22f949 P ccbe2ab276d0422eb2346623ae7c4c58: Deleting consensus metadata
W20260924 04:12:27.872836 19031 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33299 (error 108)
I20260924 04:12:27.873107 19534 ts_tablet_manager.cc:1905] T 0fb8d89024bc4cb09165e4bebb22f949 P e3951ac2843a4200adb2daf3f828b1d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:12:27.873363 19028 catalog_manager.cc:4841] TS ccbe2ab276d0422eb2346623ae7c4c58 (127.0.98.3:40737): tablet 0fb8d89024bc4cb09165e4bebb22f949 (table test-table [id=477bfd94d577425e93bdcc5b771cec38]) successfully deleted
I20260924 04:12:27.873584 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 19235
I20260924 04:12:27.875061 19509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:27.874771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33299 (local address 127.0.0.1:60082)
0924 04:12:27.874884 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:27.874962 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260924 04:12:27.875262 19031 catalog_manager.cc:4584] TS ae9a55925f484c29bbb51565adaacc53 (127.0.98.1:33299): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0fb8d89024bc4cb09165e4bebb22f949: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33299: connect: Connection refused (error 111)
I20260924 04:12:27.879484 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 19366
I20260924 04:12:27.881616 19509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:27.879553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45409 (local address 127.0.0.1:48658)
0924 04:12:27.881492 (+ 1939us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:27.881551 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:45409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1842}
I20260924 04:12:27.885334 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 19013
2026-09-24T04:12:27Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1188 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222898807720-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:12:27 dist-test-slave-00ld krb5kdc[19540](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:12:27 dist-test-slave-00ld krb5kdc[19540](info): set up 2 sockets
Sep 24 04:12:27 dist-test-slave-00ld krb5kdc[19540](info): commencing operation
krb5kdc: starting...
W20260924 04:12:28.916656 392 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222898807720-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222898807720-392-0/krb5kdc/test-user.keytab.
Sep 24 04:12:28 dist-test-slave-00ld krb5kdc[19540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223148, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:12:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:12:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:12:29.039377 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222898807720-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222898807720-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222898807720-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43245
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43245
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:12:29.146745 19553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:12:29.147004 19553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:12:29.147042 19553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:12:29.150923 19553 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:12:29.150974 19553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:12:29.150993 19553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:12:29.151010 19553 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:12:29.151024 19553 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:12:29.156042 19553 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222898807720-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222898807720-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43245
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19553
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222898807720-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:12:29.156985 19553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:12:29.157899 19553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:12:29.166440 19558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:12:29.166489 19553 server_base.cc:1034] running on GCE node
W20260924 04:12:29.166610 19561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:12:29.166510 19559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:12:29.167260 19553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:12:29.168277 19553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:12:29.169432 19553 hybrid_clock.cc:648] HybridClock initialized: now 1790223149169402 us; error 42 us; skew 500 ppm
Sep 24 04:12:29 dist-test-slave-00ld krb5kdc[19540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223149, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:12:29.172119 19553 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:12:29.173382 19553 webserver.cc:460] Webserver started at http://127.0.98.62:46373/ using document root <none> and password file <none>
I20260924 04:12:29.173952 19553 fs_manager.cc:359] Metadata directory not provided
I20260924 04:12:29.174012 19553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:12:29.174247 19553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:12:29.176278 19553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222898807720-392-0/minicluster-data/master-0/data/instance:
uuid: "ee77aeee07324f72b336ac91b3071952"
format_stamp: "Formatted at 2026-09-24 04:12:29 on dist-test-slave-00ld"
server_key: "4ad6b92ee43fe60723352f4f65f17a36"
server_key_iv: "f1d511a5257678e434764af43ac0864d"
server_key_version: "encryptionkey@0"
I20260924 04:12:29.176867 19553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222898807720-392-0/minicluster-data/master-0/wal/instance:
uuid: "ee77aeee07324f72b336ac91b3071952"
format_stamp: "Formatted at 2026-09-24 04:12:29 on dist-test-slave-00ld"
server_key: "4ad6b92ee43fe60723352f4f65f17a36"
server_key_iv: "f1d511a5257678e434764af43ac0864d"
server_key_version: "encryptionkey@0"
I20260924 04:12:29.180419 19553 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:12:29.182858 19568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:12:29.184098 19553 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:12:29.184243 19553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222898807720-392-0/minicluster-data/master-0/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222898807720-392-0/minicluster-data/master-0/wal
uuid: "ee77aeee07324f72b336ac91b3071952"
format_stamp: "Formatted at 2026-09-24 04:12:29 on dist-test-slave-00ld"
server_key: "4ad6b92ee43fe60723352f4f65f17a36"
server_key_iv: "f1d511a5257678e434764af43ac0864d"
server_key_version: "encryptionkey@0"
I20260924 04:12:29.184343 19553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222898807720-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222898807720-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222898807720-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:12:29.212137 19553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:12:29.215557 19553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:12:29.215780 19553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:12:29.222958 19553 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43245
I20260924 04:12:29.222994 19620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43245 every 8 connection(s)
I20260924 04:12:29.223902 19553 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:12:29.225533 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 19553
I20260924 04:12:29.225690 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222898807720-392-0/minicluster-data/master-0/wal/instance
I20260924 04:12:29.226024 392 external_mini_cluster.cc:1428] Setting key 60fc9304ce15cc2d091f05654fdb501c
I20260924 04:12:29.227034 19621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:12:29.233549 19621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee77aeee07324f72b336ac91b3071952: Bootstrap starting.
I20260924 04:12:29.235646 19621 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee77aeee07324f72b336ac91b3071952: Neither blocks nor log segments found. Creating new log.
Sep 24 04:12:29 dist-test-slave-00ld krb5kdc[19540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223148, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:12:29.236368 19621 log.cc:824] T 00000000000000000000000000000000 P ee77aeee07324f72b336ac91b3071952: Log is configured to *not* fsync() on all Append() calls
I20260924 04:12:29.238274 19621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee77aeee07324f72b336ac91b3071952: No bootstrap required, opened a new log
I20260924 04:12:29.240856 19624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:29.227657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36894 (local address 127.0.98.62:43245)
0924 04:12:29.228119 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:29.228134 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:29.228170 (+ 36us) server_negotiation.cc:407] Connection header received
0924 04:12:29.229082 (+ 912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:29.229102 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:29.229703 (+ 601us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:29.229987 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:12:29.230921 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:29.231926 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:29.233122 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:29.233583 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:29.233779 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:29.236248 (+ 2469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:12:29.236254 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:12:29.236276 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:12:29.236303 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:12:29.238097 (+ 1794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:12:29.238594 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:12:29.238599 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:12:29.238603 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:12:29.238707 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:12:29.238992 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:12:29.238995 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:12:29.238997 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:12:29.239376 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:12:29.239570 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:29.239823 (+ 253us) server_negotiation.cc:299] Negotiation successful
0924 04:12:29.240102 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":141,"threads_started":1}
I20260924 04:12:29.242164 19621 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee77aeee07324f72b336ac91b3071952 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee77aeee07324f72b336ac91b3071952" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43245 } }
I20260924 04:12:29.242408 19621 raft_consensus.cc:374] T 00000000000000000000000000000000 P ee77aeee07324f72b336ac91b3071952 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:12:29.242491 19621 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee77aeee07324f72b336ac91b3071952 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee77aeee07324f72b336ac91b3071952, State: Initialized, Role: FOLLOWER
I20260924 04:12:29.242920 19621 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee77aeee07324f72b336ac91b3071952 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee77aeee07324f72b336ac91b3071952" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43245 } }
I20260924 04:12:29.243098 19621 raft_consensus.cc:388] T 00000000000000000000000000000000 P ee77aeee07324f72b336ac91b3071952 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:12:29.243177 19621 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee77aeee07324f72b336ac91b3071952 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:12:29.243271 19621 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee77aeee07324f72b336ac91b3071952 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:12:29.244149 19621 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee77aeee07324f72b336ac91b3071952 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee77aeee07324f72b336ac91b3071952" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43245 } }
I20260924 04:12:29.244432 19621 leader_election.cc:302] T 00000000000000000000000000000000 P ee77aeee07324f72b336ac91b3071952 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee77aeee07324f72b336ac91b3071952; no voters:
I20260924 04:12:29.244763 19621 leader_election.cc:288] T 00000000000000000000000000000000 P ee77aeee07324f72b336ac91b3071952 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:12:29.244843 19626 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee77aeee07324f72b336ac91b3071952 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:12:29.245111 19626 raft_consensus.cc:686] T 00000000000000000000000000000000 P ee77aeee07324f72b336ac91b3071952 [term 1 LEADER]: Becoming Leader. State: Replica: ee77aeee07324f72b336ac91b3071952, State: Running, Role: LEADER
I20260924 04:12:29.245437 19626 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee77aeee07324f72b336ac91b3071952 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee77aeee07324f72b336ac91b3071952" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43245 } }
I20260924 04:12:29.246054 19621 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee77aeee07324f72b336ac91b3071952 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:12:29.247087 19627 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee77aeee07324f72b336ac91b3071952 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ee77aeee07324f72b336ac91b3071952" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee77aeee07324f72b336ac91b3071952" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43245 } } }
I20260924 04:12:29.247243 19627 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee77aeee07324f72b336ac91b3071952 [sys.catalog]: This master's current role is: LEADER
I20260924 04:12:29.247603 19635 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:12:29.247599 19628 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee77aeee07324f72b336ac91b3071952 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee77aeee07324f72b336ac91b3071952. Latest consensus state: current_term: 1 leader_uuid: "ee77aeee07324f72b336ac91b3071952" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee77aeee07324f72b336ac91b3071952" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43245 } } }
I20260924 04:12:29.247824 19628 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee77aeee07324f72b336ac91b3071952 [sys.catalog]: This master's current role is: LEADER
I20260924 04:12:29.250337 19635 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:12:29.255470 19635 catalog_manager.cc:1347] Generated new cluster ID: e746974e05a14b90882866d3bb5cb6a5
I20260924 04:12:29.255537 19635 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:12:29.275880 19635 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:12:29.277184 19635 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:12:29.281877 19635 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ee77aeee07324f72b336ac91b3071952: Generated new TSK 0
I20260924 04:12:29.282608 19635 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222898807720-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222898807720-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:12:29.311425 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222898807720-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222898807720-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222898807720-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222898807720-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43245
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222898807720-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:12:29.426087 19647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:12:29.426375 19647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:12:29.426453 19647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:12:29.430501 19647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:12:29.430583 19647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:12:29.430676 19647 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:12:29.435778 19647 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222898807720-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222898807720-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222898807720-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222898807720-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43245
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19647
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222898807720-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:12:29.436892 19647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:12:29.437951 19647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:12:29.447541 19652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:12:29.447562 19653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:12:29.447551 19655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:12:29.447741 19647 server_base.cc:1034] running on GCE node
I20260924 04:12:29.448310 19647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:12:29.449003 19647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:12:29.450204 19647 hybrid_clock.cc:648] HybridClock initialized: now 1790223149450185 us; error 33 us; skew 500 ppm
Sep 24 04:12:29 dist-test-slave-00ld krb5kdc[19540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223149, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:12:29.452950 19647 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:12:29.453831 19647 webserver.cc:460] Webserver started at http://127.0.98.1:35121/ using document root <none> and password file <none>
I20260924 04:12:29.454339 19647 fs_manager.cc:359] Metadata directory not provided
I20260924 04:12:29.454396 19647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:12:29.454555 19647 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:12:29.456483 19647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222898807720-392-0/minicluster-data/ts-0/data/instance:
uuid: "39de5b3062454ee9a15d0beff8deeb0d"
format_stamp: "Formatted at 2026-09-24 04:12:29 on dist-test-slave-00ld"
server_key: "dbfbf939ed12b39dc9d284c47229a9f9"
server_key_iv: "89ce89de40d90ff8270f083733d1ad26"
server_key_version: "encryptionkey@0"
I20260924 04:12:29.457041 19647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222898807720-392-0/minicluster-data/ts-0/wal/instance:
uuid: "39de5b3062454ee9a15d0beff8deeb0d"
format_stamp: "Formatted at 2026-09-24 04:12:29 on dist-test-slave-00ld"
server_key: "dbfbf939ed12b39dc9d284c47229a9f9"
server_key_iv: "89ce89de40d90ff8270f083733d1ad26"
server_key_version: "encryptionkey@0"
I20260924 04:12:29.460628 19647 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:12:29.463054 19662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:12:29.464293 19647 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:12:29.464419 19647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222898807720-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222898807720-392-0/minicluster-data/ts-0/wal
uuid: "39de5b3062454ee9a15d0beff8deeb0d"
format_stamp: "Formatted at 2026-09-24 04:12:29 on dist-test-slave-00ld"
server_key: "dbfbf939ed12b39dc9d284c47229a9f9"
server_key_iv: "89ce89de40d90ff8270f083733d1ad26"
server_key_version: "encryptionkey@0"
I20260924 04:12:29.464546 19647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222898807720-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222898807720-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222898807720-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:12:29.479142 19647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:12:29.481786 19647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:12:29.481968 19647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:12:29.482518 19647 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:12:29.483284 19647 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:12:29.483330 19647 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:12:29.483389 19647 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:12:29.483420 19647 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:12:29.492211 19647 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33027
I20260924 04:12:29.492239 19775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33027 every 8 connection(s)
I20260924 04:12:29.493209 19647 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222898807720-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:12:29.497920 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 19647
I20260924 04:12:29.498075 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222898807720-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:12:29.498443 392 external_mini_cluster.cc:1428] Setting key f1d1d313c73899b7e3f8aeee580383d3
Sep 24 04:12:29 dist-test-slave-00ld krb5kdc[19540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223149, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 04:12:29.507486 19624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:29.494964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35969 (local address 127.0.98.62:43245)
0924 04:12:29.495104 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:29.495107 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:29.495904 (+ 797us) server_negotiation.cc:407] Connection header received
0924 04:12:29.496817 (+ 913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:29.496820 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:29.496883 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:29.497016 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:12:29.498399 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:29.499084 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:29.500810 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:29.501132 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:29.501225 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:29.503947 (+ 2722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:12:29.503952 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:12:29.503954 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:12:29.503986 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:12:29.505439 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:12:29.505945 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:12:29.505952 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:12:29.505975 (+ 23us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:12:29.506070 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:12:29.506414 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:12:29.506418 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:12:29.506420 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:12:29.506603 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:12:29.506710 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:29.507193 (+ 483us) server_negotiation.cc:299] Negotiation successful
0924 04:12:29.507320 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 04:12:29.508292 19778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:29.495315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43245 (local address 127.0.98.1:35969)
0924 04:12:29.495754 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:29.495796 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 04:12:29.496585 (+ 789us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:12:29.497190 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:12:29.497196 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:12:29.497542 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:12:29.498220 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:29.498241 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:29.499321 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:29.499325 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:29.500611 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:29.500626 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:29.501299 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:29.501306 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:29.501540 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:29.502219 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:12:29.502236 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:12:29.503782 (+ 1546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:12:29.505560 (+ 1778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:12:29.505566 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:12:29.505582 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:12:29.505842 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:12:29.506168 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:12:29.506171 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:12:29.506173 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:12:29.506319 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:12:29.506706 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:12:29.506713 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:12:29.506959 (+ 246us) client_negotiation.cc:769] Sending connection context
0924 04:12:29.507143 (+ 184us) client_negotiation.cc:240] Negotiation successful
0924 04:12:29.507472 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":108,"threads_started":1}
I20260924 04:12:29.509533 19776 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43245
I20260924 04:12:29.509800 19776 heartbeater.cc:464] Registering TS with master...
I20260924 04:12:29.510350 19776 heartbeater.cc:511] Master 127.0.98.62:43245 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 04:12:29.512015 19585 ts_manager.cc:194] Registered new tserver with Master: 39de5b3062454ee9a15d0beff8deeb0d (127.0.98.1:33027)
I20260924 04:12:29.513203 19585 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35969
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222898807720-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222898807720-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:12:29.527899 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222898807720-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222898807720-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222898807720-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222898807720-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43245
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222898807720-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:12:29.636626 19781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:12:29.636888 19781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:12:29.636929 19781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:12:29.640839 19781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:12:29.640897 19781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:12:29.640951 19781 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:12:29.646193 19781 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222898807720-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222898807720-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222898807720-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222898807720-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43245
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19781
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222898807720-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:12:29.647264 19781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:12:29.648307 19781 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:12:29.657516 19789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:12:29.657541 19786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:12:29.657476 19787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:12:29.657752 19781 server_base.cc:1034] running on GCE node
I20260924 04:12:29.658270 19781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:12:29.658902 19781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:12:29.660089 19781 hybrid_clock.cc:648] HybridClock initialized: now 1790223149660079 us; error 27 us; skew 500 ppm
Sep 24 04:12:29 dist-test-slave-00ld krb5kdc[19540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223149, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:12:29.663765 19781 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:12:29.664726 19781 webserver.cc:460] Webserver started at http://127.0.98.2:37509/ using document root <none> and password file <none>
I20260924 04:12:29.665298 19781 fs_manager.cc:359] Metadata directory not provided
I20260924 04:12:29.665360 19781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:12:29.665525 19781 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:12:29.667497 19781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222898807720-392-0/minicluster-data/ts-1/data/instance:
uuid: "7389cf17d3664163b97455e110c478e6"
format_stamp: "Formatted at 2026-09-24 04:12:29 on dist-test-slave-00ld"
server_key: "ab0bf2dc67a1872883660ec5b8309b65"
server_key_iv: "be9f3abc8bb6459e649ab5fee746a1e2"
server_key_version: "encryptionkey@0"
I20260924 04:12:29.668066 19781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222898807720-392-0/minicluster-data/ts-1/wal/instance:
uuid: "7389cf17d3664163b97455e110c478e6"
format_stamp: "Formatted at 2026-09-24 04:12:29 on dist-test-slave-00ld"
server_key: "ab0bf2dc67a1872883660ec5b8309b65"
server_key_iv: "be9f3abc8bb6459e649ab5fee746a1e2"
server_key_version: "encryptionkey@0"
I20260924 04:12:29.671641 19781 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 04:12:29.673975 19796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:12:29.675128 19781 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:12:29.675266 19781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222898807720-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222898807720-392-0/minicluster-data/ts-1/wal
uuid: "7389cf17d3664163b97455e110c478e6"
format_stamp: "Formatted at 2026-09-24 04:12:29 on dist-test-slave-00ld"
server_key: "ab0bf2dc67a1872883660ec5b8309b65"
server_key_iv: "be9f3abc8bb6459e649ab5fee746a1e2"
server_key_version: "encryptionkey@0"
I20260924 04:12:29.675370 19781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222898807720-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222898807720-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222898807720-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:12:29.705744 19781 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:12:29.709142 19781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:12:29.709354 19781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:12:29.709945 19781 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:12:29.710772 19781 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:12:29.710841 19781 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:12:29.710913 19781 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:12:29.710965 19781 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:12:29.719949 19781 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35055
I20260924 04:12:29.719972 19909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35055 every 8 connection(s)
I20260924 04:12:29.720886 19781 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222898807720-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:12:29.724296 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 19781
I20260924 04:12:29.724457 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222898807720-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:12:29.724785 392 external_mini_cluster.cc:1428] Setting key 8121d8f64d8bad02a94c24ef921ab14f
Sep 24 04:12:29 dist-test-slave-00ld krb5kdc[19540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223149, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 04:12:29.734156 19624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:29.722620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45477 (local address 127.0.98.62:43245)
0924 04:12:29.722753 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:29.722757 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:29.723560 (+ 803us) server_negotiation.cc:407] Connection header received
0924 04:12:29.724388 (+ 828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:29.724391 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:29.724444 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:29.724528 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:12:29.726030 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:29.726792 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:29.727952 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:29.728175 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:29.728292 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:29.730716 (+ 2424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:12:29.730719 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:12:29.730721 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:12:29.730744 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:12:29.732144 (+ 1400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:12:29.732663 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:12:29.732668 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:12:29.732670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:12:29.732787 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:12:29.733123 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:12:29.733126 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:12:29.733128 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:12:29.733306 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:12:29.733410 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:29.733876 (+ 466us) server_negotiation.cc:299] Negotiation successful
0924 04:12:29.733993 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 04:12:29.735013 19912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:29.723007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43245 (local address 127.0.98.2:45477)
0924 04:12:29.723413 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:29.723452 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 04:12:29.724184 (+ 732us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:12:29.724751 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:12:29.724760 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:12:29.725225 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:12:29.725870 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:29.725888 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:29.726914 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:29.726916 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:29.727825 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:29.727835 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:29.728324 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:29.728328 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:29.728508 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:29.729134 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:12:29.729149 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:12:29.730553 (+ 1404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:12:29.732257 (+ 1704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:12:29.732265 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:12:29.732280 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:12:29.732554 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:12:29.732884 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:12:29.732887 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:12:29.732889 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:12:29.733023 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:12:29.733407 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:12:29.733413 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:12:29.733644 (+ 231us) client_negotiation.cc:769] Sending connection context
0924 04:12:29.733835 (+ 191us) client_negotiation.cc:240] Negotiation successful
0924 04:12:29.734131 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":105,"threads_started":1}
I20260924 04:12:29.736147 19910 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43245
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260924 04:12:29.736433 19910 heartbeater.cc:464] Registering TS with master...
I20260924 04:12:29.736961 19910 heartbeater.cc:511] Master 127.0.98.62:43245 requested a full tablet report, sending...
I20260924 04:12:29.737994 19585 ts_manager.cc:194] Registered new tserver with Master: 7389cf17d3664163b97455e110c478e6 (127.0.98.2:35055)
I20260924 04:12:29.738548 19585 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:45477
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222898807720-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222898807720-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:12:29.751757 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222898807720-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222898807720-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222898807720-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222898807720-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43245
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222898807720-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:12:29.862490 19915 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:12:29.862749 19915 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:12:29.862792 19915 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:12:29.866856 19915 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:12:29.866916 19915 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:12:29.867002 19915 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:12:29.872397 19915 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222898807720-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222898807720-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222898807720-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222898807720-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43245
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19915
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222898807720-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:12:29.873492 19915 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:12:29.874552 19915 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:12:29.884384 19923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:12:29.884341 19921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:12:29.884547 19915 server_base.cc:1034] running on GCE node
W20260924 04:12:29.884335 19920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:12:29.885102 19915 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:12:29.885788 19915 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:12:29.886950 19915 hybrid_clock.cc:648] HybridClock initialized: now 1790223149886934 us; error 34 us; skew 500 ppm
Sep 24 04:12:29 dist-test-slave-00ld krb5kdc[19540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223149, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:12:29.889955 19915 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:12:29.891132 19915 webserver.cc:460] Webserver started at http://127.0.98.3:40853/ using document root <none> and password file <none>
I20260924 04:12:29.891705 19915 fs_manager.cc:359] Metadata directory not provided
I20260924 04:12:29.891793 19915 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:12:29.891999 19915 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:12:29.894249 19915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222898807720-392-0/minicluster-data/ts-2/data/instance:
uuid: "d4517fe2e7ca4e529c125a3e0e2e5773"
format_stamp: "Formatted at 2026-09-24 04:12:29 on dist-test-slave-00ld"
server_key: "b935ad6579d8c920e8c31550232adbac"
server_key_iv: "8ca9ea13fe0b36e7e67cdebb067ebf0e"
server_key_version: "encryptionkey@0"
I20260924 04:12:29.894801 19915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222898807720-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d4517fe2e7ca4e529c125a3e0e2e5773"
format_stamp: "Formatted at 2026-09-24 04:12:29 on dist-test-slave-00ld"
server_key: "b935ad6579d8c920e8c31550232adbac"
server_key_iv: "8ca9ea13fe0b36e7e67cdebb067ebf0e"
server_key_version: "encryptionkey@0"
I20260924 04:12:29.898550 19915 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 04:12:29.900925 19930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:12:29.902097 19915 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:12:29.902227 19915 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222898807720-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222898807720-392-0/minicluster-data/ts-2/wal
uuid: "d4517fe2e7ca4e529c125a3e0e2e5773"
format_stamp: "Formatted at 2026-09-24 04:12:29 on dist-test-slave-00ld"
server_key: "b935ad6579d8c920e8c31550232adbac"
server_key_iv: "8ca9ea13fe0b36e7e67cdebb067ebf0e"
server_key_version: "encryptionkey@0"
I20260924 04:12:29.902349 19915 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222898807720-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222898807720-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222898807720-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:12:29.919687 19915 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:12:29.922968 19915 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:12:29.923169 19915 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:12:29.923759 19915 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:12:29.924549 19915 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:12:29.924623 19915 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:12:29.924688 19915 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:12:29.924768 19915 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:12:29.933945 19915 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39719
I20260924 04:12:29.933972 20043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39719 every 8 connection(s)
I20260924 04:12:29.934840 19915 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222898807720-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:12:29.938023 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 19915
I20260924 04:12:29.938162 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222898807720-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:12:29.938444 392 external_mini_cluster.cc:1428] Setting key 931f874f53f2e30ac2e93f7a0900f186
Sep 24 04:12:29 dist-test-slave-00ld krb5kdc[19540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223149, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:12:29.948882 19624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:29.936583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50551 (local address 127.0.98.62:43245)
0924 04:12:29.936767 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:29.936771 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:29.937755 (+ 984us) server_negotiation.cc:407] Connection header received
0924 04:12:29.938817 (+ 1062us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:29.938820 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:29.938869 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:29.938975 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:12:29.940521 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:29.941147 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:29.942264 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:29.942491 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:29.942580 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:29.945119 (+ 2539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:12:29.945122 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:12:29.945124 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:12:29.945164 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:12:29.946661 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:12:29.947297 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:12:29.947303 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:12:29.947307 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:12:29.947393 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:12:29.947815 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:12:29.947817 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:12:29.947818 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:12:29.947983 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:12:29.948057 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:29.948594 (+ 537us) server_negotiation.cc:299] Negotiation successful
0924 04:12:29.948759 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 04:12:29.949790 20046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:29.936958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43245 (local address 127.0.98.3:50551)
0924 04:12:29.937595 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:29.937635 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 04:12:29.938586 (+ 951us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:12:29.939189 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:12:29.939198 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:12:29.939709 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:12:29.940377 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:29.940395 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:29.941256 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:29.941259 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:29.942131 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:29.942140 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:29.942636 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:29.942639 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:29.942781 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:29.943366 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:12:29.943382 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:12:29.944969 (+ 1587us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:12:29.946811 (+ 1842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:12:29.946815 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:12:29.946827 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:12:29.947193 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:12:29.947553 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:12:29.947559 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:12:29.947564 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:12:29.947727 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:12:29.948082 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:12:29.948087 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:12:29.948387 (+ 300us) client_negotiation.cc:769] Sending connection context
0924 04:12:29.948547 (+ 160us) client_negotiation.cc:240] Negotiation successful
0924 04:12:29.948873 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":105,"threads_started":1}
I20260924 04:12:29.951120 20044 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43245
I20260924 04:12:29.951400 20044 heartbeater.cc:464] Registering TS with master...
I20260924 04:12:29.952013 20044 heartbeater.cc:511] Master 127.0.98.62:43245 requested a full tablet report, sending...
I20260924 04:12:29.953152 19585 ts_manager.cc:194] Registered new tserver with Master: d4517fe2e7ca4e529c125a3e0e2e5773 (127.0.98.3:39719)
I20260924 04:12:29.953675 19585 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50551
I20260924 04:12:29.964393 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:12:29.973449 19624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:29.966327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39240 (local address 127.0.98.62:43245)
0924 04:12:29.966446 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:29.966449 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:29.966667 (+ 218us) server_negotiation.cc:407] Connection header received
0924 04:12:29.966770 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:29.966772 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:29.966810 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:29.966875 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:12:29.967514 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:29.968032 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:29.969003 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:29.969218 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:29.969298 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:29.970649 (+ 1351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:12:29.970652 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:12:29.970654 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:12:29.970676 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:12:29.971905 (+ 1229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:12:29.972275 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:12:29.972278 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:12:29.972279 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:12:29.972361 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:12:29.972616 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:12:29.972618 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:12:29.972620 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:12:29.972927 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:12:29.973101 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:29.973237 (+ 136us) server_negotiation.cc:299] Negotiation successful
0924 04:12:29.973336 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 04:12:29.976238 19585 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39240:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:12:29.979364 19585 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:12:29.993558 20056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:29.987385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38954 (local address 127.0.98.3:39719)
0924 04:12:29.988047 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:29.988059 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:29.988082 (+ 23us) server_negotiation.cc:407] Connection header received
0924 04:12:29.988276 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:29.988284 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:29.988512 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:29.988647 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:12:29.989160 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:29.990363 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:29.992189 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:29.992818 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:29.992931 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:29.993085 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:29.993279 (+ 194us) server_negotiation.cc:299] Negotiation successful
0924 04:12:29.993392 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":109,"threads_started":1}
I20260924 04:12:29.993664 20059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:29.988003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35055 (local address 127.0.0.1:47566)
0924 04:12:29.988777 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:29.988789 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:12:29.988870 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:12:29.989566 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:12:29.989569 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:12:29.989587 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:12:29.989679 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:29.989685 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:29.991148 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:29.991154 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:29.992141 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:29.992152 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:29.993196 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:29.993202 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:29.993296 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:29.993318 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 04:12:29.993388 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 04:12:29.993509 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":710,"mutex_wait_us":64,"thread_start_us":102,"threads_started":1}
I20260924 04:12:29.993696 20060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:29.988420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47566 (local address 127.0.98.2:35055)
0924 04:12:29.989130 (+ 710us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:29.989142 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:29.989161 (+ 19us) server_negotiation.cc:407] Connection header received
0924 04:12:29.989229 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:29.989234 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:29.989403 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:29.989818 (+ 415us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:12:29.989960 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:29.991005 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:29.992287 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:29.993032 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:29.993172 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:29.993289 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:29.993429 (+ 140us) server_negotiation.cc:299] Negotiation successful
0924 04:12:29.993526 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":94,"threads_started":1}
I20260924 04:12:29.994001 20058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:29.987601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41210 (local address 127.0.98.1:33027)
0924 04:12:29.988269 (+ 668us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:29.988284 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:29.988804 (+ 520us) server_negotiation.cc:407] Connection header received
0924 04:12:29.988886 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:29.988890 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:29.989091 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:29.989240 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:12:29.989452 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:29.990171 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:29.992157 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:29.993024 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:29.993146 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:29.993289 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:29.993748 (+ 459us) server_negotiation.cc:299] Negotiation successful
0924 04:12:29.993855 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":104,"threads_started":1}
I20260924 04:12:29.994242 20055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:29.987444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33027 (local address 127.0.0.1:41210)
0924 04:12:29.988574 (+ 1130us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:29.988589 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:12:29.988664 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:12:29.989236 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:12:29.989239 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:12:29.989255 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:12:29.989353 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:29.989362 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:29.990330 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:29.990336 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:29.992022 (+ 1686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:29.992035 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:29.993356 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:29.993380 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:29.993473 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:29.993496 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 04:12:29.993578 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 04:12:29.993652 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1060,"mutex_wait_us":165,"thread_start_us":78,"threads_started":1}
I20260924 04:12:29.994851 20057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:29.987286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39719 (local address 127.0.0.1:38954)
0924 04:12:29.987912 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:29.987960 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 04:12:29.988085 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:12:29.988886 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:12:29.988892 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:12:29.988916 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:12:29.989039 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:29.989045 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:29.990489 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:29.990492 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:29.992089 (+ 1597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:29.992099 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:29.992941 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:29.992945 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:29.993049 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:29.993086 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 04:12:29.994674 (+ 1588us) client_negotiation.cc:240] Negotiation successful
0924 04:12:29.994760 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"mutex_wait_us":128,"thread_start_us":81,"threads_started":1}
I20260924 04:12:29.996951 19710 tablet_service.cc:1467] Processing CreateTablet for tablet 1678739ff287450f9a7b9eb4392f8c3b (DEFAULT_TABLE table=test-table [id=f63a5ed3e5b7448a804d3a5cbbe31a5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:12:29.997076 19844 tablet_service.cc:1467] Processing CreateTablet for tablet 1678739ff287450f9a7b9eb4392f8c3b (DEFAULT_TABLE table=test-table [id=f63a5ed3e5b7448a804d3a5cbbe31a5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:12:29.997305 19978 tablet_service.cc:1467] Processing CreateTablet for tablet 1678739ff287450f9a7b9eb4392f8c3b (DEFAULT_TABLE table=test-table [id=f63a5ed3e5b7448a804d3a5cbbe31a5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:12:29.998046 19710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1678739ff287450f9a7b9eb4392f8c3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:12:29.998083 19844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1678739ff287450f9a7b9eb4392f8c3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:12:29.998111 19978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1678739ff287450f9a7b9eb4392f8c3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:12:30.003926 20061 tablet_bootstrap.cc:492] T 1678739ff287450f9a7b9eb4392f8c3b P 39de5b3062454ee9a15d0beff8deeb0d: Bootstrap starting.
I20260924 04:12:30.005322 20062 tablet_bootstrap.cc:492] T 1678739ff287450f9a7b9eb4392f8c3b P 7389cf17d3664163b97455e110c478e6: Bootstrap starting.
I20260924 04:12:30.005949 20063 tablet_bootstrap.cc:492] T 1678739ff287450f9a7b9eb4392f8c3b P d4517fe2e7ca4e529c125a3e0e2e5773: Bootstrap starting.
I20260924 04:12:30.006366 20061 tablet_bootstrap.cc:654] T 1678739ff287450f9a7b9eb4392f8c3b P 39de5b3062454ee9a15d0beff8deeb0d: Neither blocks nor log segments found. Creating new log.
I20260924 04:12:30.007650 20061 log.cc:824] T 1678739ff287450f9a7b9eb4392f8c3b P 39de5b3062454ee9a15d0beff8deeb0d: Log is configured to *not* fsync() on all Append() calls
I20260924 04:12:30.008183 20063 tablet_bootstrap.cc:654] T 1678739ff287450f9a7b9eb4392f8c3b P d4517fe2e7ca4e529c125a3e0e2e5773: Neither blocks nor log segments found. Creating new log.
I20260924 04:12:30.009420 20063 log.cc:824] T 1678739ff287450f9a7b9eb4392f8c3b P d4517fe2e7ca4e529c125a3e0e2e5773: Log is configured to *not* fsync() on all Append() calls
I20260924 04:12:30.009747 20061 tablet_bootstrap.cc:492] T 1678739ff287450f9a7b9eb4392f8c3b P 39de5b3062454ee9a15d0beff8deeb0d: No bootstrap required, opened a new log
I20260924 04:12:30.009974 20061 ts_tablet_manager.cc:1397] T 1678739ff287450f9a7b9eb4392f8c3b P 39de5b3062454ee9a15d0beff8deeb0d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 04:12:30.010439 20062 tablet_bootstrap.cc:654] T 1678739ff287450f9a7b9eb4392f8c3b P 7389cf17d3664163b97455e110c478e6: Neither blocks nor log segments found. Creating new log.
I20260924 04:12:30.011185 20062 log.cc:824] T 1678739ff287450f9a7b9eb4392f8c3b P 7389cf17d3664163b97455e110c478e6: Log is configured to *not* fsync() on all Append() calls
I20260924 04:12:30.011394 20063 tablet_bootstrap.cc:492] T 1678739ff287450f9a7b9eb4392f8c3b P d4517fe2e7ca4e529c125a3e0e2e5773: No bootstrap required, opened a new log
I20260924 04:12:30.011567 20063 ts_tablet_manager.cc:1397] T 1678739ff287450f9a7b9eb4392f8c3b P d4517fe2e7ca4e529c125a3e0e2e5773: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260924 04:12:30.013116 20062 tablet_bootstrap.cc:492] T 1678739ff287450f9a7b9eb4392f8c3b P 7389cf17d3664163b97455e110c478e6: No bootstrap required, opened a new log
I20260924 04:12:30.013293 20062 ts_tablet_manager.cc:1397] T 1678739ff287450f9a7b9eb4392f8c3b P 7389cf17d3664163b97455e110c478e6: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.004s
I20260924 04:12:30.013401 20061 raft_consensus.cc:348] T 1678739ff287450f9a7b9eb4392f8c3b P 39de5b3062454ee9a15d0beff8deeb0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4517fe2e7ca4e529c125a3e0e2e5773" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39719 } } peers { permanent_uuid: "7389cf17d3664163b97455e110c478e6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35055 } } peers { permanent_uuid: "39de5b3062454ee9a15d0beff8deeb0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33027 } }
I20260924 04:12:30.013643 20061 raft_consensus.cc:374] T 1678739ff287450f9a7b9eb4392f8c3b P 39de5b3062454ee9a15d0beff8deeb0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:12:30.013693 20061 raft_consensus.cc:729] T 1678739ff287450f9a7b9eb4392f8c3b P 39de5b3062454ee9a15d0beff8deeb0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39de5b3062454ee9a15d0beff8deeb0d, State: Initialized, Role: FOLLOWER
I20260924 04:12:30.014132 20061 consensus_queue.cc:260] T 1678739ff287450f9a7b9eb4392f8c3b P 39de5b3062454ee9a15d0beff8deeb0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4517fe2e7ca4e529c125a3e0e2e5773" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39719 } } peers { permanent_uuid: "7389cf17d3664163b97455e110c478e6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35055 } } peers { permanent_uuid: "39de5b3062454ee9a15d0beff8deeb0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33027 } }
I20260924 04:12:30.014714 19776 heartbeater.cc:503] Master 127.0.98.62:43245 was elected leader, sending a full tablet report...
I20260924 04:12:30.015074 20061 ts_tablet_manager.cc:1428] T 1678739ff287450f9a7b9eb4392f8c3b P 39de5b3062454ee9a15d0beff8deeb0d: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20260924 04:12:30.015364 20063 raft_consensus.cc:348] T 1678739ff287450f9a7b9eb4392f8c3b P d4517fe2e7ca4e529c125a3e0e2e5773 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4517fe2e7ca4e529c125a3e0e2e5773" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39719 } } peers { permanent_uuid: "7389cf17d3664163b97455e110c478e6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35055 } } peers { permanent_uuid: "39de5b3062454ee9a15d0beff8deeb0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33027 } }
I20260924 04:12:30.015612 20063 raft_consensus.cc:374] T 1678739ff287450f9a7b9eb4392f8c3b P d4517fe2e7ca4e529c125a3e0e2e5773 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:12:30.015659 20063 raft_consensus.cc:729] T 1678739ff287450f9a7b9eb4392f8c3b P d4517fe2e7ca4e529c125a3e0e2e5773 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4517fe2e7ca4e529c125a3e0e2e5773, State: Initialized, Role: FOLLOWER
I20260924 04:12:30.016116 20063 consensus_queue.cc:260] T 1678739ff287450f9a7b9eb4392f8c3b P d4517fe2e7ca4e529c125a3e0e2e5773 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4517fe2e7ca4e529c125a3e0e2e5773" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39719 } } peers { permanent_uuid: "7389cf17d3664163b97455e110c478e6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35055 } } peers { permanent_uuid: "39de5b3062454ee9a15d0beff8deeb0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33027 } }
I20260924 04:12:30.017046 20063 ts_tablet_manager.cc:1428] T 1678739ff287450f9a7b9eb4392f8c3b P d4517fe2e7ca4e529c125a3e0e2e5773: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260924 04:12:30.017074 20044 heartbeater.cc:503] Master 127.0.98.62:43245 was elected leader, sending a full tablet report...
I20260924 04:12:30.017030 20062 raft_consensus.cc:348] T 1678739ff287450f9a7b9eb4392f8c3b P 7389cf17d3664163b97455e110c478e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4517fe2e7ca4e529c125a3e0e2e5773" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39719 } } peers { permanent_uuid: "7389cf17d3664163b97455e110c478e6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35055 } } peers { permanent_uuid: "39de5b3062454ee9a15d0beff8deeb0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33027 } }
I20260924 04:12:30.017293 20062 raft_consensus.cc:374] T 1678739ff287450f9a7b9eb4392f8c3b P 7389cf17d3664163b97455e110c478e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:12:30.017333 20062 raft_consensus.cc:729] T 1678739ff287450f9a7b9eb4392f8c3b P 7389cf17d3664163b97455e110c478e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7389cf17d3664163b97455e110c478e6, State: Initialized, Role: FOLLOWER
I20260924 04:12:30.017669 20062 consensus_queue.cc:260] T 1678739ff287450f9a7b9eb4392f8c3b P 7389cf17d3664163b97455e110c478e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4517fe2e7ca4e529c125a3e0e2e5773" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39719 } } peers { permanent_uuid: "7389cf17d3664163b97455e110c478e6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35055 } } peers { permanent_uuid: "39de5b3062454ee9a15d0beff8deeb0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33027 } }
I20260924 04:12:30.018520 20062 ts_tablet_manager.cc:1428] T 1678739ff287450f9a7b9eb4392f8c3b P 7389cf17d3664163b97455e110c478e6: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260924 04:12:30.018839 19910 heartbeater.cc:503] Master 127.0.98.62:43245 was elected leader, sending a full tablet report...
I20260924 04:12:30.047204 20068 raft_consensus.cc:482] T 1678739ff287450f9a7b9eb4392f8c3b P d4517fe2e7ca4e529c125a3e0e2e5773 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:12:30.047492 20068 raft_consensus.cc:504] T 1678739ff287450f9a7b9eb4392f8c3b P d4517fe2e7ca4e529c125a3e0e2e5773 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4517fe2e7ca4e529c125a3e0e2e5773" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39719 } } peers { permanent_uuid: "7389cf17d3664163b97455e110c478e6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35055 } } peers { permanent_uuid: "39de5b3062454ee9a15d0beff8deeb0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33027 } }
I20260924 04:12:30.048942 20068 leader_election.cc:288] T 1678739ff287450f9a7b9eb4392f8c3b P d4517fe2e7ca4e529c125a3e0e2e5773 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7389cf17d3664163b97455e110c478e6 (127.0.98.2:35055), 39de5b3062454ee9a15d0beff8deeb0d (127.0.98.1:33027)
I20260924 04:12:30.052615 20046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:30.049102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35055 (local address 127.0.98.3:55191)
0924 04:12:30.049423 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:30.049438 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:12:30.049535 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:12:30.049840 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:12:30.049842 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:12:30.049857 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:12:30.049956 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:30.049961 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:30.050721 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:30.050724 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:30.051513 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:30.051522 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:30.052179 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:30.052182 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:30.052287 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:30.052315 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 04:12:30.052405 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 04:12:30.052479 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"mutex_wait_us":1}
I20260924 04:12:30.052609 20060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:30.049270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55191 (local address 127.0.98.2:35055)
0924 04:12:30.049412 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:30.049416 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:30.049484 (+ 68us) server_negotiation.cc:407] Connection header received
0924 04:12:30.049668 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:30.049671 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:30.049730 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:30.049810 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:12:30.050074 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:30.050583 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:30.051641 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:30.052055 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:30.052186 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:30.052250 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:30.052419 (+ 169us) server_negotiation.cc:299] Negotiation successful
0924 04:12:30.052482 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 04:12:30.052862 20058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:30.049645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55823 (local address 127.0.98.1:33027)
0924 04:12:30.049960 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:30.049963 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:30.049974 (+ 11us) server_negotiation.cc:407] Connection header received
0924 04:12:30.050018 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:30.050020 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:30.050052 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:30.050138 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:12:30.050417 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:30.050907 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:30.051940 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:30.052321 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:30.052437 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:30.052496 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:30.052662 (+ 166us) server_negotiation.cc:299] Negotiation successful
0924 04:12:30.052752 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236}
I20260924 04:12:30.052881 20070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:30.049207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33027 (local address 127.0.98.3:55823)
0924 04:12:30.049544 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:30.049559 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:12:30.049680 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:12:30.050170 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:12:30.050173 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:12:30.050188 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:12:30.050304 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:30.050309 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:30.051020 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:30.051025 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:30.051818 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:30.051826 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:30.052440 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:30.052444 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:30.052537 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:30.052563 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 04:12:30.052647 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 04:12:30.052739 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"mutex_wait_us":22,"thread_start_us":134,"threads_started":1}
I20260924 04:12:30.053391 19864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1678739ff287450f9a7b9eb4392f8c3b" candidate_uuid: "d4517fe2e7ca4e529c125a3e0e2e5773" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7389cf17d3664163b97455e110c478e6" is_pre_election: true
I20260924 04:12:30.053531 19730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1678739ff287450f9a7b9eb4392f8c3b" candidate_uuid: "d4517fe2e7ca4e529c125a3e0e2e5773" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39de5b3062454ee9a15d0beff8deeb0d" is_pre_election: true
I20260924 04:12:30.053725 19864 raft_consensus.cc:2454] T 1678739ff287450f9a7b9eb4392f8c3b P 7389cf17d3664163b97455e110c478e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4517fe2e7ca4e529c125a3e0e2e5773 in term 0.
I20260924 04:12:30.053886 19730 raft_consensus.cc:2454] T 1678739ff287450f9a7b9eb4392f8c3b P 39de5b3062454ee9a15d0beff8deeb0d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4517fe2e7ca4e529c125a3e0e2e5773 in term 0.
I20260924 04:12:30.054306 19933 leader_election.cc:302] T 1678739ff287450f9a7b9eb4392f8c3b P d4517fe2e7ca4e529c125a3e0e2e5773 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7389cf17d3664163b97455e110c478e6, d4517fe2e7ca4e529c125a3e0e2e5773; no voters:
I20260924 04:12:30.054603 20068 raft_consensus.cc:2781] T 1678739ff287450f9a7b9eb4392f8c3b P d4517fe2e7ca4e529c125a3e0e2e5773 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:12:30.054685 20068 raft_consensus.cc:482] T 1678739ff287450f9a7b9eb4392f8c3b P d4517fe2e7ca4e529c125a3e0e2e5773 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:12:30.054740 20068 raft_consensus.cc:3037] T 1678739ff287450f9a7b9eb4392f8c3b P d4517fe2e7ca4e529c125a3e0e2e5773 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:12:30.055727 20068 raft_consensus.cc:504] T 1678739ff287450f9a7b9eb4392f8c3b P d4517fe2e7ca4e529c125a3e0e2e5773 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4517fe2e7ca4e529c125a3e0e2e5773" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39719 } } peers { permanent_uuid: "7389cf17d3664163b97455e110c478e6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35055 } } peers { permanent_uuid: "39de5b3062454ee9a15d0beff8deeb0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33027 } }
I20260924 04:12:30.056087 20068 leader_election.cc:288] T 1678739ff287450f9a7b9eb4392f8c3b P d4517fe2e7ca4e529c125a3e0e2e5773 [CANDIDATE]: Term 1 election: Requested vote from peers 7389cf17d3664163b97455e110c478e6 (127.0.98.2:35055), 39de5b3062454ee9a15d0beff8deeb0d (127.0.98.1:33027)
I20260924 04:12:30.056480 19864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1678739ff287450f9a7b9eb4392f8c3b" candidate_uuid: "d4517fe2e7ca4e529c125a3e0e2e5773" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7389cf17d3664163b97455e110c478e6"
I20260924 04:12:30.056496 19730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1678739ff287450f9a7b9eb4392f8c3b" candidate_uuid: "d4517fe2e7ca4e529c125a3e0e2e5773" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39de5b3062454ee9a15d0beff8deeb0d"
I20260924 04:12:30.056582 19864 raft_consensus.cc:3037] T 1678739ff287450f9a7b9eb4392f8c3b P 7389cf17d3664163b97455e110c478e6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:12:30.056629 19730 raft_consensus.cc:3037] T 1678739ff287450f9a7b9eb4392f8c3b P 39de5b3062454ee9a15d0beff8deeb0d [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:12:30.057638 19864 raft_consensus.cc:2454] T 1678739ff287450f9a7b9eb4392f8c3b P 7389cf17d3664163b97455e110c478e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4517fe2e7ca4e529c125a3e0e2e5773 in term 1.
I20260924 04:12:30.057770 19730 raft_consensus.cc:2454] T 1678739ff287450f9a7b9eb4392f8c3b P 39de5b3062454ee9a15d0beff8deeb0d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4517fe2e7ca4e529c125a3e0e2e5773 in term 1.
I20260924 04:12:30.057956 19933 leader_election.cc:302] T 1678739ff287450f9a7b9eb4392f8c3b P d4517fe2e7ca4e529c125a3e0e2e5773 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7389cf17d3664163b97455e110c478e6, d4517fe2e7ca4e529c125a3e0e2e5773; no voters:
I20260924 04:12:30.058131 20068 raft_consensus.cc:2781] T 1678739ff287450f9a7b9eb4392f8c3b P d4517fe2e7ca4e529c125a3e0e2e5773 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:12:30.058374 20068 raft_consensus.cc:686] T 1678739ff287450f9a7b9eb4392f8c3b P d4517fe2e7ca4e529c125a3e0e2e5773 [term 1 LEADER]: Becoming Leader. State: Replica: d4517fe2e7ca4e529c125a3e0e2e5773, State: Running, Role: LEADER
I20260924 04:12:30.058698 20068 consensus_queue.cc:237] T 1678739ff287450f9a7b9eb4392f8c3b P d4517fe2e7ca4e529c125a3e0e2e5773 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4517fe2e7ca4e529c125a3e0e2e5773" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39719 } } peers { permanent_uuid: "7389cf17d3664163b97455e110c478e6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35055 } } peers { permanent_uuid: "39de5b3062454ee9a15d0beff8deeb0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33027 } }
I20260924 04:12:30.061887 19584 catalog_manager.cc:5495] T 1678739ff287450f9a7b9eb4392f8c3b P d4517fe2e7ca4e529c125a3e0e2e5773 reported cstate change: term changed from 0 to 1, leader changed from <none> to d4517fe2e7ca4e529c125a3e0e2e5773 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "d4517fe2e7ca4e529c125a3e0e2e5773" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4517fe2e7ca4e529c125a3e0e2e5773" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7389cf17d3664163b97455e110c478e6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39de5b3062454ee9a15d0beff8deeb0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33027 } health_report { overall_health: UNKNOWN } } }
I20260924 04:12:30.090679 20056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:30.087020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38956 (local address 127.0.98.3:39719)
0924 04:12:30.087169 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:30.087174 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:30.087188 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:12:30.087338 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:30.087341 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:30.087394 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:30.087502 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:12:30.087765 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:30.088271 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:30.089205 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:30.089453 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:30.089562 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:30.089864 (+ 302us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:12:30.090271 (+ 407us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:12:30.090404 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:30.090463 (+ 59us) server_negotiation.cc:299] Negotiation successful
0924 04:12:30.090517 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 04:12:30.096455 19864 raft_consensus.cc:1261] T 1678739ff287450f9a7b9eb4392f8c3b P 7389cf17d3664163b97455e110c478e6 [term 1 FOLLOWER]: Refusing update from remote peer d4517fe2e7ca4e529c125a3e0e2e5773: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:12:30.096457 19730 raft_consensus.cc:1261] T 1678739ff287450f9a7b9eb4392f8c3b P 39de5b3062454ee9a15d0beff8deeb0d [term 1 FOLLOWER]: Refusing update from remote peer d4517fe2e7ca4e529c125a3e0e2e5773: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:12:30.097270 20071 consensus_queue.cc:1035] T 1678739ff287450f9a7b9eb4392f8c3b P d4517fe2e7ca4e529c125a3e0e2e5773 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7389cf17d3664163b97455e110c478e6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:12:30.097438 20068 consensus_queue.cc:1035] T 1678739ff287450f9a7b9eb4392f8c3b P d4517fe2e7ca4e529c125a3e0e2e5773 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39de5b3062454ee9a15d0beff8deeb0d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:12:30.105528 20075 mvcc.cc:205] Tried to move back new op lower bound from 7332754022787850240 to 7332754022643298304. Current Snapshot: MvccSnapshot[applied={T|T < 7332754022787850240}]
I20260924 04:12:30.112320 19584 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39240:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:12:30.112462 19584 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39240:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:12:30.114977 19584 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ee77aeee07324f72b336ac91b3071952: Sending DeleteTablet for 3 replicas of tablet 1678739ff287450f9a7b9eb4392f8c3b
I20260924 04:12:30.115764 19978 tablet_service.cc:1514] Processing DeleteTablet for tablet 1678739ff287450f9a7b9eb4392f8c3b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:12:30 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38954
I20260924 04:12:30.115762 19710 tablet_service.cc:1514] Processing DeleteTablet for tablet 1678739ff287450f9a7b9eb4392f8c3b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:12:30 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41210
I20260924 04:12:30.115809 19844 tablet_service.cc:1514] Processing DeleteTablet for tablet 1678739ff287450f9a7b9eb4392f8c3b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:12:30 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47566
I20260924 04:12:30.116341 20084 tablet_replica.cc:330] T 1678739ff287450f9a7b9eb4392f8c3b P 7389cf17d3664163b97455e110c478e6: stopping tablet replica
I20260924 04:12:30.116658 20084 raft_consensus.cc:2229] T 1678739ff287450f9a7b9eb4392f8c3b P 7389cf17d3664163b97455e110c478e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:12:30.116981 20084 raft_consensus.cc:2258] T 1678739ff287450f9a7b9eb4392f8c3b P 7389cf17d3664163b97455e110c478e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:12:30.117079 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 19647
I20260924 04:12:30.117964 20085 tablet_replica.cc:330] T 1678739ff287450f9a7b9eb4392f8c3b P d4517fe2e7ca4e529c125a3e0e2e5773: stopping tablet replica
I20260924 04:12:30.118263 20085 raft_consensus.cc:2229] T 1678739ff287450f9a7b9eb4392f8c3b P d4517fe2e7ca4e529c125a3e0e2e5773 [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:12:30.118642 20085 raft_consensus.cc:2258] T 1678739ff287450f9a7b9eb4392f8c3b P d4517fe2e7ca4e529c125a3e0e2e5773 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:12:30.119664 20084 ts_tablet_manager.cc:1905] T 1678739ff287450f9a7b9eb4392f8c3b P 7389cf17d3664163b97455e110c478e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:12:30.120081 20085 ts_tablet_manager.cc:1905] T 1678739ff287450f9a7b9eb4392f8c3b P d4517fe2e7ca4e529c125a3e0e2e5773: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:12:30.122339 20084 ts_tablet_manager.cc:1918] T 1678739ff287450f9a7b9eb4392f8c3b P 7389cf17d3664163b97455e110c478e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:12:30.122512 20084 log.cc:1196] T 1678739ff287450f9a7b9eb4392f8c3b P 7389cf17d3664163b97455e110c478e6: Deleting WAL directory at /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222898807720-392-0/minicluster-data/ts-1/wal/wals/1678739ff287450f9a7b9eb4392f8c3b
I20260924 04:12:30.122910 20084 ts_tablet_manager.cc:1939] T 1678739ff287450f9a7b9eb4392f8c3b P 7389cf17d3664163b97455e110c478e6: Deleting consensus metadata
I20260924 04:12:30.123482 20085 ts_tablet_manager.cc:1918] T 1678739ff287450f9a7b9eb4392f8c3b P d4517fe2e7ca4e529c125a3e0e2e5773: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:12:30.123581 20085 log.cc:1196] T 1678739ff287450f9a7b9eb4392f8c3b P d4517fe2e7ca4e529c125a3e0e2e5773: Deleting WAL directory at /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790222898807720-392-0/minicluster-data/ts-2/wal/wals/1678739ff287450f9a7b9eb4392f8c3b
I20260924 04:12:30.123909 20085 ts_tablet_manager.cc:1939] T 1678739ff287450f9a7b9eb4392f8c3b P d4517fe2e7ca4e529c125a3e0e2e5773: Deleting consensus metadata
I20260924 04:12:30.124442 19571 catalog_manager.cc:4841] TS 7389cf17d3664163b97455e110c478e6 (127.0.98.2:35055): tablet 1678739ff287450f9a7b9eb4392f8c3b (table test-table [id=f63a5ed3e5b7448a804d3a5cbbe31a5a]) successfully deleted
W20260924 04:12:30.125226 19569 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33027 (error 108)
I20260924 04:12:30.125411 19571 catalog_manager.cc:4841] TS d4517fe2e7ca4e529c125a3e0e2e5773 (127.0.98.3:39719): tablet 1678739ff287450f9a7b9eb4392f8c3b (table test-table [id=f63a5ed3e5b7448a804d3a5cbbe31a5a]) successfully deleted
I20260924 04:12:30.126070 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 19781
I20260924 04:12:30.126298 20055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:30.125986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33027 (local address 127.0.0.1:41212)
0924 04:12:30.126149 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:30.126212 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
W20260924 04:12:30.126755 19569 catalog_manager.cc:4584] TS 39de5b3062454ee9a15d0beff8deeb0d (127.0.98.1:33027): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1678739ff287450f9a7b9eb4392f8c3b: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33027: connect: Connection refused (error 111)
I20260924 04:12:30.131925 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 19915
I20260924 04:12:30.137036 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 19553
2026-09-24T04:12:30Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2252 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-24T04:12:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:12:30Z Disabled control of system clock
I20260924 04:12:30.167021 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222898807720-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222898807720-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222898807720-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41247
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:39329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41247
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 04:12:30.277371 20091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:12:30.277621 20091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:12:30.277660 20091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:12:30.281605 20091 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:12:30.281663 20091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:12:30.281678 20091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:12:30.281695 20091 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:12:30.281709 20091 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:12:30.286346 20091 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222898807720-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222898807720-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41247
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41247
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20091
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222898807720-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:12:30.287283 20091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:12:30.288535 20091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:12:30.296983 20099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:12:30.296994 20096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:12:30.296990 20097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:12:30.297065 20091 server_base.cc:1034] running on GCE node
I20260924 04:12:30.297744 20091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:12:30.298825 20091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:12:30.300006 20091 hybrid_clock.cc:648] HybridClock initialized: now 1790223150299994 us; error 29 us; skew 500 ppm
I20260924 04:12:30.301855 20091 webserver.cc:460] Webserver started at http://127.0.98.62:33693/ using document root <none> and password file <none>
I20260924 04:12:30.302429 20091 fs_manager.cc:359] Metadata directory not provided
I20260924 04:12:30.302523 20091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:12:30.302755 20091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:12:30.304886 20091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222898807720-392-0/minicluster-data/master-0/data/instance:
uuid: "7eef76e1cdb645b8bd8996a6e879d01d"
format_stamp: "Formatted at 2026-09-24 04:12:30 on dist-test-slave-00ld"
server_key: "6ed5e29d7c488f93f6eb6e452ed06e20"
server_key_iv: "dac56f8bf8a272e0ad696bd85e10636a"
server_key_version: "encryptionkey@0"
I20260924 04:12:30.305434 20091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222898807720-392-0/minicluster-data/master-0/wal/instance:
uuid: "7eef76e1cdb645b8bd8996a6e879d01d"
format_stamp: "Formatted at 2026-09-24 04:12:30 on dist-test-slave-00ld"
server_key: "6ed5e29d7c488f93f6eb6e452ed06e20"
server_key_iv: "dac56f8bf8a272e0ad696bd85e10636a"
server_key_version: "encryptionkey@0"
I20260924 04:12:30.309027 20091 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:12:30.311702 20105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:12:30.312922 20091 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:12:30.313045 20091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222898807720-392-0/minicluster-data/master-0/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222898807720-392-0/minicluster-data/master-0/wal
uuid: "7eef76e1cdb645b8bd8996a6e879d01d"
format_stamp: "Formatted at 2026-09-24 04:12:30 on dist-test-slave-00ld"
server_key: "6ed5e29d7c488f93f6eb6e452ed06e20"
server_key_iv: "dac56f8bf8a272e0ad696bd85e10636a"
server_key_version: "encryptionkey@0"
I20260924 04:12:30.313167 20091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222898807720-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222898807720-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222898807720-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:12:30.347560 20091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:12:30.348294 20091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:12:30.348510 20091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:12:30.355430 20091 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41247
I20260924 04:12:30.355443 20157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41247 every 8 connection(s)
I20260924 04:12:30.356408 20091 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:12:30.359504 20158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:12:30.362694 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 20091
I20260924 04:12:30.362807 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222898807720-392-0/minicluster-data/master-0/wal/instance
I20260924 04:12:30.363096 392 external_mini_cluster.cc:1428] Setting key 44ffc8b75662a5b9dcc1446f04fa440a
I20260924 04:12:30.365553 20158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7eef76e1cdb645b8bd8996a6e879d01d: Bootstrap starting.
I20260924 04:12:30.367885 20158 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7eef76e1cdb645b8bd8996a6e879d01d: Neither blocks nor log segments found. Creating new log.
I20260924 04:12:30.368631 20158 log.cc:824] T 00000000000000000000000000000000 P 7eef76e1cdb645b8bd8996a6e879d01d: Log is configured to *not* fsync() on all Append() calls
I20260924 04:12:30.370599 20158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7eef76e1cdb645b8bd8996a6e879d01d: No bootstrap required, opened a new log
I20260924 04:12:30.372591 20161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:30.364540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40936 (local address 127.0.98.62:41247)
0924 04:12:30.365210 (+ 670us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:30.365258 (+ 48us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:30.365304 (+ 46us) server_negotiation.cc:407] Connection header received
0924 04:12:30.366284 (+ 980us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:30.366303 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:30.366700 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:30.367035 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:12:30.367468 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:30.368482 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:30.369466 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:30.369941 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:30.370109 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:30.370967 (+ 858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:12:30.370972 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:12:30.370985 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:12:30.371365 (+ 380us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:12:30.371419 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:12:30.371438 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:12:30.371543 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:30.371661 (+ 118us) server_negotiation.cc:299] Negotiation successful
0924 04:12:30.371915 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":144,"threads_started":1}
I20260924 04:12:30.374455 20158 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7eef76e1cdb645b8bd8996a6e879d01d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eef76e1cdb645b8bd8996a6e879d01d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41247 } }
I20260924 04:12:30.374754 20158 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7eef76e1cdb645b8bd8996a6e879d01d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:12:30.374815 20158 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7eef76e1cdb645b8bd8996a6e879d01d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eef76e1cdb645b8bd8996a6e879d01d, State: Initialized, Role: FOLLOWER
I20260924 04:12:30.375255 20158 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7eef76e1cdb645b8bd8996a6e879d01d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eef76e1cdb645b8bd8996a6e879d01d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41247 } }
I20260924 04:12:30.375411 20158 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7eef76e1cdb645b8bd8996a6e879d01d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:12:30.375474 20158 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7eef76e1cdb645b8bd8996a6e879d01d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:12:30.375617 20158 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7eef76e1cdb645b8bd8996a6e879d01d [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:12:30.376607 20158 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7eef76e1cdb645b8bd8996a6e879d01d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eef76e1cdb645b8bd8996a6e879d01d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41247 } }
I20260924 04:12:30.376950 20158 leader_election.cc:302] T 00000000000000000000000000000000 P 7eef76e1cdb645b8bd8996a6e879d01d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7eef76e1cdb645b8bd8996a6e879d01d; no voters:
I20260924 04:12:30.377256 20158 leader_election.cc:288] T 00000000000000000000000000000000 P 7eef76e1cdb645b8bd8996a6e879d01d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:12:30.377364 20163 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7eef76e1cdb645b8bd8996a6e879d01d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:12:30.377588 20163 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7eef76e1cdb645b8bd8996a6e879d01d [term 1 LEADER]: Becoming Leader. State: Replica: 7eef76e1cdb645b8bd8996a6e879d01d, State: Running, Role: LEADER
I20260924 04:12:30.377892 20163 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7eef76e1cdb645b8bd8996a6e879d01d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eef76e1cdb645b8bd8996a6e879d01d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41247 } }
I20260924 04:12:30.378490 20158 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7eef76e1cdb645b8bd8996a6e879d01d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:12:30.379498 20165 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7eef76e1cdb645b8bd8996a6e879d01d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7eef76e1cdb645b8bd8996a6e879d01d. Latest consensus state: current_term: 1 leader_uuid: "7eef76e1cdb645b8bd8996a6e879d01d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eef76e1cdb645b8bd8996a6e879d01d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41247 } } }
I20260924 04:12:30.379633 20165 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7eef76e1cdb645b8bd8996a6e879d01d [sys.catalog]: This master's current role is: LEADER
I20260924 04:12:30.379902 20164 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7eef76e1cdb645b8bd8996a6e879d01d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7eef76e1cdb645b8bd8996a6e879d01d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eef76e1cdb645b8bd8996a6e879d01d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41247 } } }
I20260924 04:12:30.379994 20164 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7eef76e1cdb645b8bd8996a6e879d01d [sys.catalog]: This master's current role is: LEADER
I20260924 04:12:30.380095 20172 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:12:30.383106 20172 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:12:30.389213 20172 catalog_manager.cc:1347] Generated new cluster ID: ffa75fbbe13e4d699556c0472619db10
I20260924 04:12:30.389300 20172 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:12:30.403360 20172 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:12:30.404578 20172 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:12:30.425181 20172 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7eef76e1cdb645b8bd8996a6e879d01d: Generated new TSK 0
I20260924 04:12:30.426056 20172 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:12:30.432331 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222898807720-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222898807720-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222898807720-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222898807720-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41247
--builtin_ntp_servers=127.0.98.20:39329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 04:12:30.544247 20182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:12:30.544552 20182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:12:30.544622 20182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:12:30.548745 20182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:12:30.548827 20182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:12:30.548914 20182 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:12:30.553715 20182 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222898807720-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222898807720-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222898807720-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20182
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222898807720-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:12:30.554790 20182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:12:30.556200 20182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:12:30.565497 20190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:12:30.565565 20188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:12:30.565623 20182 server_base.cc:1034] running on GCE node
W20260924 04:12:30.565497 20187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:12:30.566236 20182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:12:30.566875 20182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:12:30.568053 20182 hybrid_clock.cc:648] HybridClock initialized: now 1790223150568031 us; error 31 us; skew 500 ppm
I20260924 04:12:30.569832 20182 webserver.cc:460] Webserver started at http://127.0.98.1:43489/ using document root <none> and password file <none>
I20260924 04:12:30.570402 20182 fs_manager.cc:359] Metadata directory not provided
I20260924 04:12:30.570490 20182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:12:30.570747 20182 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:12:30.573032 20182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222898807720-392-0/minicluster-data/ts-0/data/instance:
uuid: "8d2142cdc3cf4b6eb6b4def03be15d93"
format_stamp: "Formatted at 2026-09-24 04:12:30 on dist-test-slave-00ld"
server_key: "ced83933db37195a4917d80e62f1c9bd"
server_key_iv: "e7630c7f2a153b08d9eb7e208120ead2"
server_key_version: "encryptionkey@0"
I20260924 04:12:30.573606 20182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222898807720-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8d2142cdc3cf4b6eb6b4def03be15d93"
format_stamp: "Formatted at 2026-09-24 04:12:30 on dist-test-slave-00ld"
server_key: "ced83933db37195a4917d80e62f1c9bd"
server_key_iv: "e7630c7f2a153b08d9eb7e208120ead2"
server_key_version: "encryptionkey@0"
I20260924 04:12:30.577263 20182 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:12:30.579809 20196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:12:30.581187 20182 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:12:30.581293 20182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222898807720-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222898807720-392-0/minicluster-data/ts-0/wal
uuid: "8d2142cdc3cf4b6eb6b4def03be15d93"
format_stamp: "Formatted at 2026-09-24 04:12:30 on dist-test-slave-00ld"
server_key: "ced83933db37195a4917d80e62f1c9bd"
server_key_iv: "e7630c7f2a153b08d9eb7e208120ead2"
server_key_version: "encryptionkey@0"
I20260924 04:12:30.581385 20182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222898807720-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222898807720-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222898807720-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:12:30.605185 20182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:12:30.605856 20182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:12:30.606017 20182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:12:30.606588 20182 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:12:30.607512 20182 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:12:30.607565 20182 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:12:30.607615 20182 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:12:30.607640 20182 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:12:30.617318 20182 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:32831
I20260924 04:12:30.617342 20309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:32831 every 8 connection(s)
I20260924 04:12:30.618305 20182 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222898807720-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:12:30.618750 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 20182
I20260924 04:12:30.618907 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222898807720-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:12:30.619251 392 external_mini_cluster.cc:1428] Setting key e4f21319f11d3370633df22448dbe397
I20260924 04:12:30.621512 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222898807720-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222898807720-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222898807720-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222898807720-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41247
--builtin_ntp_servers=127.0.98.20:39329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 04:12:30.628775 20161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:30.620661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57751 (local address 127.0.98.62:41247)
0924 04:12:30.620821 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:30.620826 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:30.621782 (+ 956us) server_negotiation.cc:407] Connection header received
0924 04:12:30.623179 (+ 1397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:30.623184 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:30.623252 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:30.623349 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:12:30.624260 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:30.625203 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:30.626459 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:30.626900 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:30.627031 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:30.628171 (+ 1140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:12:30.628178 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:12:30.628183 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:12:30.628201 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:12:30.628228 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:12:30.628239 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:12:30.628328 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:30.628466 (+ 138us) server_negotiation.cc:299] Negotiation successful
0924 04:12:30.628541 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 04:12:30.629585 20312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:30.620815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41247 (local address 127.0.98.1:57751)
0924 04:12:30.621560 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:30.621658 (+ 98us) client_negotiation.cc:174] Beginning negotiation
0924 04:12:30.622968 (+ 1310us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:12:30.623501 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:12:30.623508 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:12:30.623600 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:12:30.624075 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:30.624128 (+ 53us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:30.625340 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:30.625343 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:30.626327 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:30.626337 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:30.627068 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:30.627070 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:30.627246 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:30.627910 (+ 664us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:12:30.627912 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:12:30.627950 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:12:30.628010 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:12:30.628330 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:12:30.628337 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:12:30.628340 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 04:12:30.628432 (+ 92us) client_negotiation.cc:240] Negotiation successful
0924 04:12:30.628640 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":134,"threads_started":1}
I20260924 04:12:30.630962 20310 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41247
I20260924 04:12:30.631189 20310 heartbeater.cc:464] Registering TS with master...
I20260924 04:12:30.631722 20310 heartbeater.cc:511] Master 127.0.98.62:41247 requested a full tablet report, sending...
I20260924 04:12:30.633339 20122 ts_manager.cc:194] Registered new tserver with Master: 8d2142cdc3cf4b6eb6b4def03be15d93 (127.0.98.1:32831)
I20260924 04:12:30.634829 20122 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:57751
W20260924 04:12:30.733796 20313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:12:30.734042 20313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:12:30.734078 20313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:12:30.738077 20313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:12:30.738142 20313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:12:30.738195 20313 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:12:30.742897 20313 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222898807720-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222898807720-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222898807720-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20313
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222898807720-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:12:30.743942 20313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:12:30.745290 20313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:12:30.754956 20321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:12:30.755069 20318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:12:30.754983 20319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:12:30.755151 20313 server_base.cc:1034] running on GCE node
I20260924 04:12:30.755647 20313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:12:30.756342 20313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:12:30.757521 20313 hybrid_clock.cc:648] HybridClock initialized: now 1790223150757504 us; error 30 us; skew 500 ppm
I20260924 04:12:30.759352 20313 webserver.cc:460] Webserver started at http://127.0.98.2:41311/ using document root <none> and password file <none>
I20260924 04:12:30.759935 20313 fs_manager.cc:359] Metadata directory not provided
I20260924 04:12:30.760027 20313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:12:30.760262 20313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:12:30.762353 20313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222898807720-392-0/minicluster-data/ts-1/data/instance:
uuid: "b93c03ff9c7b43618ca45d7c398ecc01"
format_stamp: "Formatted at 2026-09-24 04:12:30 on dist-test-slave-00ld"
server_key: "fa4f5001525ef2ec563d94839444c96b"
server_key_iv: "72ec94d471528c6e364af3ba0a6e595e"
server_key_version: "encryptionkey@0"
I20260924 04:12:30.762936 20313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222898807720-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b93c03ff9c7b43618ca45d7c398ecc01"
format_stamp: "Formatted at 2026-09-24 04:12:30 on dist-test-slave-00ld"
server_key: "fa4f5001525ef2ec563d94839444c96b"
server_key_iv: "72ec94d471528c6e364af3ba0a6e595e"
server_key_version: "encryptionkey@0"
I20260924 04:12:30.766562 20313 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260924 04:12:30.769078 20327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:12:30.770283 20313 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 04:12:30.770413 20313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222898807720-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222898807720-392-0/minicluster-data/ts-1/wal
uuid: "b93c03ff9c7b43618ca45d7c398ecc01"
format_stamp: "Formatted at 2026-09-24 04:12:30 on dist-test-slave-00ld"
server_key: "fa4f5001525ef2ec563d94839444c96b"
server_key_iv: "72ec94d471528c6e364af3ba0a6e595e"
server_key_version: "encryptionkey@0"
I20260924 04:12:30.770530 20313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222898807720-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222898807720-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222898807720-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:12:30.783156 20313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:12:30.783774 20313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:12:30.783958 20313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:12:30.784552 20313 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:12:30.785531 20313 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:12:30.785598 20313 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:12:30.785668 20313 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:12:30.785719 20313 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:12:30.794672 20313 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37051
I20260924 04:12:30.794697 20440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37051 every 8 connection(s)
I20260924 04:12:30.795518 20313 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222898807720-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:12:30.797900 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 20313
I20260924 04:12:30.798029 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222898807720-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:12:30.798349 392 external_mini_cluster.cc:1428] Setting key d0657a2b7874d8c67c17bea9be6ee341
I20260924 04:12:30.800326 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222898807720-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222898807720-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222898807720-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222898807720-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41247
--builtin_ntp_servers=127.0.98.20:39329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 04:12:30.805660 20161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:30.797418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40225 (local address 127.0.98.62:41247)
0924 04:12:30.797598 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:30.797602 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:30.798890 (+ 1288us) server_negotiation.cc:407] Connection header received
0924 04:12:30.800635 (+ 1745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:30.800642 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:30.800736 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:30.800830 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:12:30.802017 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:30.802531 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:30.803642 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:30.803850 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:30.803957 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:30.805173 (+ 1216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:12:30.805177 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:12:30.805179 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:12:30.805192 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:12:30.805217 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:12:30.805225 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:12:30.805300 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:30.805456 (+ 156us) server_negotiation.cc:299] Negotiation successful
0924 04:12:30.805512 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 04:12:30.806460 20443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:30.797882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41247 (local address 127.0.98.2:40225)
0924 04:12:30.798723 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:30.798781 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 04:12:30.800361 (+ 1580us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:12:30.801041 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:12:30.801048 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:12:30.801188 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:12:30.801850 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:30.801872 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:30.802661 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:30.802667 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:30.803537 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:30.803545 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:30.804005 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:30.804010 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:30.804184 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:30.804911 (+ 727us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:12:30.804913 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:12:30.804951 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:12:30.805009 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:12:30.805314 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:12:30.805322 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:12:30.805328 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 04:12:30.805456 (+ 128us) client_negotiation.cc:240] Negotiation successful
0924 04:12:30.805723 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":140,"threads_started":1}
I20260924 04:12:30.807694 20441 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41247
I20260924 04:12:30.807912 20441 heartbeater.cc:464] Registering TS with master...
I20260924 04:12:30.808389 20441 heartbeater.cc:511] Master 127.0.98.62:41247 requested a full tablet report, sending...
I20260924 04:12:30.809441 20122 ts_manager.cc:194] Registered new tserver with Master: b93c03ff9c7b43618ca45d7c398ecc01 (127.0.98.2:37051)
I20260924 04:12:30.809983 20122 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:40225
W20260924 04:12:30.924958 20444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:12:30.925372 20444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:12:30.925458 20444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:12:30.933851 20444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:12:30.933933 20444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:12:30.934055 20444 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:12:30.944242 20444 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222898807720-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222898807720-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222898807720-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20444
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222898807720-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:12:30.945676 20444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:12:30.947032 20444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:12:30.956730 20452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:12:30.956753 20449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:12:30.956809 20450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:12:30.956997 20444 server_base.cc:1034] running on GCE node
I20260924 04:12:30.957425 20444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:12:30.958088 20444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:12:30.959280 20444 hybrid_clock.cc:648] HybridClock initialized: now 1790223150959262 us; error 31 us; skew 500 ppm
I20260924 04:12:30.961340 20444 webserver.cc:460] Webserver started at http://127.0.98.3:32941/ using document root <none> and password file <none>
I20260924 04:12:30.961951 20444 fs_manager.cc:359] Metadata directory not provided
I20260924 04:12:30.962045 20444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:12:30.962289 20444 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:12:30.964337 20444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222898807720-392-0/minicluster-data/ts-2/data/instance:
uuid: "19835f8931d54b468a37bf3b79546223"
format_stamp: "Formatted at 2026-09-24 04:12:30 on dist-test-slave-00ld"
server_key: "f3f7d647dc9251f20a54725ec92f0fe1"
server_key_iv: "02889053d3ac73b42630d69d2866e305"
server_key_version: "encryptionkey@0"
I20260924 04:12:30.964944 20444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222898807720-392-0/minicluster-data/ts-2/wal/instance:
uuid: "19835f8931d54b468a37bf3b79546223"
format_stamp: "Formatted at 2026-09-24 04:12:30 on dist-test-slave-00ld"
server_key: "f3f7d647dc9251f20a54725ec92f0fe1"
server_key_iv: "02889053d3ac73b42630d69d2866e305"
server_key_version: "encryptionkey@0"
I20260924 04:12:30.968734 20444 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 04:12:30.971217 20458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:12:30.972369 20444 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:12:30.972500 20444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222898807720-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222898807720-392-0/minicluster-data/ts-2/wal
uuid: "19835f8931d54b468a37bf3b79546223"
format_stamp: "Formatted at 2026-09-24 04:12:30 on dist-test-slave-00ld"
server_key: "f3f7d647dc9251f20a54725ec92f0fe1"
server_key_iv: "02889053d3ac73b42630d69d2866e305"
server_key_version: "encryptionkey@0"
I20260924 04:12:30.972609 20444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222898807720-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222898807720-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222898807720-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:12:30.981909 20444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:12:30.982509 20444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:12:30.982688 20444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:12:30.983206 20444 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:12:30.984074 20444 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:12:30.984144 20444 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:12:30.984212 20444 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:12:30.984270 20444 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:12:30.993245 20444 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38103
I20260924 04:12:30.993263 20571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38103 every 8 connection(s)
I20260924 04:12:30.994180 20444 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222898807720-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:12:30.997779 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 20444
I20260924 04:12:30.997911 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222898807720-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:12:30.998189 392 external_mini_cluster.cc:1428] Setting key d9ddfc6df6b87bd8207e5874e30525cb
I20260924 04:12:31.003819 20161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:30.996065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60353 (local address 127.0.98.62:41247)
0924 04:12:30.996235 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:30.996238 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:30.997262 (+ 1024us) server_negotiation.cc:407] Connection header received
0924 04:12:30.998798 (+ 1536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:30.998802 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:30.998857 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:30.998936 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:12:30.999879 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:31.000648 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:31.001854 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:31.002077 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:31.002195 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:31.003323 (+ 1128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:12:31.003327 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:12:31.003329 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:12:31.003349 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:12:31.003368 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:12:31.003375 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:12:31.003458 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:31.003619 (+ 161us) server_negotiation.cc:299] Negotiation successful
0924 04:12:31.003679 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260924 04:12:31.004640 20574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:30.996456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41247 (local address 127.0.98.3:60353)
0924 04:12:30.997106 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:30.997145 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 04:12:30.998593 (+ 1448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:12:30.999106 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:12:30.999115 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:12:30.999205 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:12:30.999739 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:30.999752 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:31.000831 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:31.000835 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:31.001726 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:31.001735 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:31.002243 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:31.002246 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:31.002410 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:31.003041 (+ 631us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:12:31.003042 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:12:31.003082 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:12:31.003157 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:12:31.003460 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:12:31.003471 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:12:31.003478 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 04:12:31.003619 (+ 141us) client_negotiation.cc:240] Negotiation successful
0924 04:12:31.003898 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":109,"threads_started":1}
I20260924 04:12:31.005749 20572 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41247
I20260924 04:12:31.005982 20572 heartbeater.cc:464] Registering TS with master...
I20260924 04:12:31.006494 20572 heartbeater.cc:511] Master 127.0.98.62:41247 requested a full tablet report, sending...
I20260924 04:12:31.007457 20122 ts_manager.cc:194] Registered new tserver with Master: 19835f8931d54b468a37bf3b79546223 (127.0.98.3:38103)
I20260924 04:12:31.007966 20122 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:60353
I20260924 04:12:31.011912 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:12:31.017766 20161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:31.013810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40948 (local address 127.0.98.62:41247)
0924 04:12:31.013957 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:31.013963 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:31.014158 (+ 195us) server_negotiation.cc:407] Connection header received
0924 04:12:31.014307 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:31.014310 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:31.014350 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:31.014436 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:12:31.014710 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:31.015217 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:31.016152 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:31.016382 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:31.016486 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:31.017361 (+ 875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:12:31.017365 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:12:31.017368 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:12:31.017380 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:12:31.017398 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:12:31.017409 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:12:31.017489 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:31.017579 (+ 90us) server_negotiation.cc:299] Negotiation successful
0924 04:12:31.017632 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 04:12:31.020406 20122 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40948:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:12:31.023481 20122 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:12:31.038085 20588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:31.033002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39120 (local address 127.0.98.3:38103)
0924 04:12:31.033484 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:31.033494 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:31.033511 (+ 17us) server_negotiation.cc:407] Connection header received
0924 04:12:31.033567 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:31.033570 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:31.033764 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:31.033923 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:12:31.034129 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:31.034846 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:31.036822 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:31.037369 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:31.037496 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:31.037709 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:31.037776 (+ 67us) server_negotiation.cc:299] Negotiation successful
0924 04:12:31.037880 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":128,"threads_started":1}
I20260924 04:12:31.038079 20586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:31.032565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38103 (local address 127.0.0.1:39120)
0924 04:12:31.033177 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:31.033190 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:12:31.033288 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:12:31.033906 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:12:31.033909 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:12:31.033926 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:12:31.034002 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:31.034008 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:31.034958 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:31.034964 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:31.036644 (+ 1680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:31.036651 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:31.037497 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:31.037500 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:31.037606 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:31.037642 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 04:12:31.037748 (+ 106us) client_negotiation.cc:240] Negotiation successful
0924 04:12:31.037884 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"mutex_wait_us":139,"thread_start_us":87,"threads_started":1}
I20260924 04:12:31.038111 20583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:31.031799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37051 (local address 127.0.0.1:39334)
0924 04:12:31.032232 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:31.032278 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 04:12:31.032408 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:12:31.033291 (+ 883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:12:31.033298 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:12:31.033319 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:12:31.033457 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:31.033463 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:31.035083 (+ 1620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:31.035086 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:31.036640 (+ 1554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:31.036651 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:31.037658 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:31.037661 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:31.037751 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:31.037774 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 04:12:31.037827 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 04:12:31.037901 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"mutex_wait_us":304,"thread_start_us":85,"threads_started":1}
I20260924 04:12:31.038455 20584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:31.031876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32831 (local address 127.0.0.1:60122)
0924 04:12:31.033036 (+ 1160us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:31.033059 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 04:12:31.033153 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:12:31.033666 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:12:31.033669 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:12:31.033686 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:12:31.033789 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:31.033795 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:31.035643 (+ 1848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:31.035646 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:31.036467 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:31.036476 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:31.037843 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:31.037847 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:31.037949 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:31.037969 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 04:12:31.038034 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 04:12:31.038093 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1101,"mutex_wait_us":2,"thread_start_us":95,"threads_started":1}
I20260924 04:12:31.038869 20585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:31.031917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39334 (local address 127.0.98.2:37051)
0924 04:12:31.032767 (+ 850us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:31.032782 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:31.032810 (+ 28us) server_negotiation.cc:407] Connection header received
0924 04:12:31.032890 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:31.032895 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:31.033124 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:31.033625 (+ 501us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:12:31.033761 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:31.034970 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:31.036778 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:31.037529 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:31.038186 (+ 657us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:31.038535 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:31.038586 (+ 51us) server_negotiation.cc:299] Negotiation successful
0924 04:12:31.038685 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":699,"thread_start_us":98,"threads_started":1}
I20260924 04:12:31.039618 20587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:31.032598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60122 (local address 127.0.98.1:32831)
0924 04:12:31.033017 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:31.033028 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:31.033257 (+ 229us) server_negotiation.cc:407] Connection header received
0924 04:12:31.033336 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:31.033340 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:31.033523 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:31.033657 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:12:31.034151 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:31.035521 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:31.036615 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:31.037453 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:31.037587 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:31.037786 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:31.039260 (+ 1474us) server_negotiation.cc:299] Negotiation successful
0924 04:12:31.039368 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":111,"threads_started":1}
I20260924 04:12:31.041749 20375 tablet_service.cc:1467] Processing CreateTablet for tablet 0fd7c227ea4e4b1cba69109c7c5b6ddd (DEFAULT_TABLE table=test-table [id=3f8618fa890b4170a8dfa6cafd7849df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:12:31.041754 20506 tablet_service.cc:1467] Processing CreateTablet for tablet 0fd7c227ea4e4b1cba69109c7c5b6ddd (DEFAULT_TABLE table=test-table [id=3f8618fa890b4170a8dfa6cafd7849df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:12:31.041993 20244 tablet_service.cc:1467] Processing CreateTablet for tablet 0fd7c227ea4e4b1cba69109c7c5b6ddd (DEFAULT_TABLE table=test-table [id=3f8618fa890b4170a8dfa6cafd7849df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:12:31.042932 20375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fd7c227ea4e4b1cba69109c7c5b6ddd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:12:31.042932 20506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fd7c227ea4e4b1cba69109c7c5b6ddd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:12:31.042981 20244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fd7c227ea4e4b1cba69109c7c5b6ddd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:12:31.049166 20589 tablet_bootstrap.cc:492] T 0fd7c227ea4e4b1cba69109c7c5b6ddd P 8d2142cdc3cf4b6eb6b4def03be15d93: Bootstrap starting.
I20260924 04:12:31.050631 20590 tablet_bootstrap.cc:492] T 0fd7c227ea4e4b1cba69109c7c5b6ddd P 19835f8931d54b468a37bf3b79546223: Bootstrap starting.
I20260924 04:12:31.051311 20589 tablet_bootstrap.cc:654] T 0fd7c227ea4e4b1cba69109c7c5b6ddd P 8d2142cdc3cf4b6eb6b4def03be15d93: Neither blocks nor log segments found. Creating new log.
I20260924 04:12:31.052016 20589 log.cc:824] T 0fd7c227ea4e4b1cba69109c7c5b6ddd P 8d2142cdc3cf4b6eb6b4def03be15d93: Log is configured to *not* fsync() on all Append() calls
I20260924 04:12:31.052393 20590 tablet_bootstrap.cc:654] T 0fd7c227ea4e4b1cba69109c7c5b6ddd P 19835f8931d54b468a37bf3b79546223: Neither blocks nor log segments found. Creating new log.
I20260924 04:12:31.052673 20591 tablet_bootstrap.cc:492] T 0fd7c227ea4e4b1cba69109c7c5b6ddd P b93c03ff9c7b43618ca45d7c398ecc01: Bootstrap starting.
I20260924 04:12:31.053220 20590 log.cc:824] T 0fd7c227ea4e4b1cba69109c7c5b6ddd P 19835f8931d54b468a37bf3b79546223: Log is configured to *not* fsync() on all Append() calls
I20260924 04:12:31.054108 20589 tablet_bootstrap.cc:492] T 0fd7c227ea4e4b1cba69109c7c5b6ddd P 8d2142cdc3cf4b6eb6b4def03be15d93: No bootstrap required, opened a new log
I20260924 04:12:31.054296 20589 ts_tablet_manager.cc:1397] T 0fd7c227ea4e4b1cba69109c7c5b6ddd P 8d2142cdc3cf4b6eb6b4def03be15d93: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 04:12:31.054733 20591 tablet_bootstrap.cc:654] T 0fd7c227ea4e4b1cba69109c7c5b6ddd P b93c03ff9c7b43618ca45d7c398ecc01: Neither blocks nor log segments found. Creating new log.
I20260924 04:12:31.054989 20590 tablet_bootstrap.cc:492] T 0fd7c227ea4e4b1cba69109c7c5b6ddd P 19835f8931d54b468a37bf3b79546223: No bootstrap required, opened a new log
I20260924 04:12:31.055176 20590 ts_tablet_manager.cc:1397] T 0fd7c227ea4e4b1cba69109c7c5b6ddd P 19835f8931d54b468a37bf3b79546223: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 04:12:31.055485 20591 log.cc:824] T 0fd7c227ea4e4b1cba69109c7c5b6ddd P b93c03ff9c7b43618ca45d7c398ecc01: Log is configured to *not* fsync() on all Append() calls
I20260924 04:12:31.057392 20591 tablet_bootstrap.cc:492] T 0fd7c227ea4e4b1cba69109c7c5b6ddd P b93c03ff9c7b43618ca45d7c398ecc01: No bootstrap required, opened a new log
I20260924 04:12:31.057571 20591 ts_tablet_manager.cc:1397] T 0fd7c227ea4e4b1cba69109c7c5b6ddd P b93c03ff9c7b43618ca45d7c398ecc01: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 04:12:31.058637 20590 raft_consensus.cc:348] T 0fd7c227ea4e4b1cba69109c7c5b6ddd P 19835f8931d54b468a37bf3b79546223 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b93c03ff9c7b43618ca45d7c398ecc01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37051 } } peers { permanent_uuid: "8d2142cdc3cf4b6eb6b4def03be15d93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32831 } } peers { permanent_uuid: "19835f8931d54b468a37bf3b79546223" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38103 } }
I20260924 04:12:31.058686 20589 raft_consensus.cc:348] T 0fd7c227ea4e4b1cba69109c7c5b6ddd P 8d2142cdc3cf4b6eb6b4def03be15d93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b93c03ff9c7b43618ca45d7c398ecc01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37051 } } peers { permanent_uuid: "8d2142cdc3cf4b6eb6b4def03be15d93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32831 } } peers { permanent_uuid: "19835f8931d54b468a37bf3b79546223" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38103 } }
I20260924 04:12:31.058876 20590 raft_consensus.cc:374] T 0fd7c227ea4e4b1cba69109c7c5b6ddd P 19835f8931d54b468a37bf3b79546223 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:12:31.058934 20589 raft_consensus.cc:374] T 0fd7c227ea4e4b1cba69109c7c5b6ddd P 8d2142cdc3cf4b6eb6b4def03be15d93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:12:31.058931 20590 raft_consensus.cc:729] T 0fd7c227ea4e4b1cba69109c7c5b6ddd P 19835f8931d54b468a37bf3b79546223 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19835f8931d54b468a37bf3b79546223, State: Initialized, Role: FOLLOWER
I20260924 04:12:31.058992 20589 raft_consensus.cc:729] T 0fd7c227ea4e4b1cba69109c7c5b6ddd P 8d2142cdc3cf4b6eb6b4def03be15d93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d2142cdc3cf4b6eb6b4def03be15d93, State: Initialized, Role: FOLLOWER
I20260924 04:12:31.059379 20590 consensus_queue.cc:260] T 0fd7c227ea4e4b1cba69109c7c5b6ddd P 19835f8931d54b468a37bf3b79546223 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b93c03ff9c7b43618ca45d7c398ecc01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37051 } } peers { permanent_uuid: "8d2142cdc3cf4b6eb6b4def03be15d93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32831 } } peers { permanent_uuid: "19835f8931d54b468a37bf3b79546223" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38103 } }
I20260924 04:12:31.059429 20589 consensus_queue.cc:260] T 0fd7c227ea4e4b1cba69109c7c5b6ddd P 8d2142cdc3cf4b6eb6b4def03be15d93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b93c03ff9c7b43618ca45d7c398ecc01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37051 } } peers { permanent_uuid: "8d2142cdc3cf4b6eb6b4def03be15d93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32831 } } peers { permanent_uuid: "19835f8931d54b468a37bf3b79546223" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38103 } }
I20260924 04:12:31.060024 20572 heartbeater.cc:503] Master 127.0.98.62:41247 was elected leader, sending a full tablet report...
I20260924 04:12:31.060379 20589 ts_tablet_manager.cc:1428] T 0fd7c227ea4e4b1cba69109c7c5b6ddd P 8d2142cdc3cf4b6eb6b4def03be15d93: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260924 04:12:31.060386 20590 ts_tablet_manager.cc:1428] T 0fd7c227ea4e4b1cba69109c7c5b6ddd P 19835f8931d54b468a37bf3b79546223: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260924 04:12:31.060729 20310 heartbeater.cc:503] Master 127.0.98.62:41247 was elected leader, sending a full tablet report...
I20260924 04:12:31.062861 20591 raft_consensus.cc:348] T 0fd7c227ea4e4b1cba69109c7c5b6ddd P b93c03ff9c7b43618ca45d7c398ecc01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b93c03ff9c7b43618ca45d7c398ecc01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37051 } } peers { permanent_uuid: "8d2142cdc3cf4b6eb6b4def03be15d93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32831 } } peers { permanent_uuid: "19835f8931d54b468a37bf3b79546223" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38103 } }
I20260924 04:12:31.063145 20591 raft_consensus.cc:374] T 0fd7c227ea4e4b1cba69109c7c5b6ddd P b93c03ff9c7b43618ca45d7c398ecc01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:12:31.063218 20591 raft_consensus.cc:729] T 0fd7c227ea4e4b1cba69109c7c5b6ddd P b93c03ff9c7b43618ca45d7c398ecc01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b93c03ff9c7b43618ca45d7c398ecc01, State: Initialized, Role: FOLLOWER
I20260924 04:12:31.063673 20591 consensus_queue.cc:260] T 0fd7c227ea4e4b1cba69109c7c5b6ddd P b93c03ff9c7b43618ca45d7c398ecc01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b93c03ff9c7b43618ca45d7c398ecc01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37051 } } peers { permanent_uuid: "8d2142cdc3cf4b6eb6b4def03be15d93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32831 } } peers { permanent_uuid: "19835f8931d54b468a37bf3b79546223" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38103 } }
I20260924 04:12:31.064412 20441 heartbeater.cc:503] Master 127.0.98.62:41247 was elected leader, sending a full tablet report...
I20260924 04:12:31.064669 20591 ts_tablet_manager.cc:1428] T 0fd7c227ea4e4b1cba69109c7c5b6ddd P b93c03ff9c7b43618ca45d7c398ecc01: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260924 04:12:31.106182 20597 raft_consensus.cc:482] T 0fd7c227ea4e4b1cba69109c7c5b6ddd P b93c03ff9c7b43618ca45d7c398ecc01 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:12:31.106426 20597 raft_consensus.cc:504] T 0fd7c227ea4e4b1cba69109c7c5b6ddd P b93c03ff9c7b43618ca45d7c398ecc01 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b93c03ff9c7b43618ca45d7c398ecc01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37051 } } peers { permanent_uuid: "8d2142cdc3cf4b6eb6b4def03be15d93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32831 } } peers { permanent_uuid: "19835f8931d54b468a37bf3b79546223" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38103 } }
I20260924 04:12:31.107649 20597 leader_election.cc:288] T 0fd7c227ea4e4b1cba69109c7c5b6ddd P b93c03ff9c7b43618ca45d7c398ecc01 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d2142cdc3cf4b6eb6b4def03be15d93 (127.0.98.1:32831), 19835f8931d54b468a37bf3b79546223 (127.0.98.3:38103)
I20260924 04:12:31.111390 20588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:31.108037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37517 (local address 127.0.98.3:38103)
0924 04:12:31.108159 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:31.108164 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:31.108180 (+ 16us) server_negotiation.cc:407] Connection header received
0924 04:12:31.108366 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:31.108369 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:31.108422 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:31.108509 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:12:31.108831 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:31.109409 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:31.110441 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:31.110842 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:31.110953 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:31.111011 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:31.111206 (+ 195us) server_negotiation.cc:299] Negotiation successful
0924 04:12:31.111271 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 04:12:31.111389 20443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:31.107905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38103 (local address 127.0.98.2:37517)
0924 04:12:31.108113 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:31.108130 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 04:12:31.108225 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:12:31.108544 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:12:31.108550 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:12:31.108575 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:12:31.108727 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:31.108739 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:31.109525 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:31.109530 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:31.110319 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:31.110327 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:31.110966 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:31.110970 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:31.111079 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:31.111103 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 04:12:31.111189 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 04:12:31.111258 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121}
I20260924 04:12:31.111845 20598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:31.107909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32831 (local address 127.0.98.2:54001)
0924 04:12:31.108514 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:31.108527 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:12:31.108696 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:12:31.109019 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:12:31.109022 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:12:31.109038 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:12:31.109194 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:31.109200 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:31.109943 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:31.109949 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:31.110716 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:31.110724 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:31.111359 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:31.111363 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:31.111466 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:31.111499 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 04:12:31.111624 (+ 125us) client_negotiation.cc:240] Negotiation successful
0924 04:12:31.111691 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"mutex_wait_us":251,"thread_start_us":136,"threads_started":1}
I20260924 04:12:31.111867 20587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:31.108103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54001 (local address 127.0.98.1:32831)
0924 04:12:31.108241 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:31.108246 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:31.108613 (+ 367us) server_negotiation.cc:407] Connection header received
0924 04:12:31.108853 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:31.108856 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:31.108911 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:31.109007 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:12:31.109308 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:31.109832 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:31.110840 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:31.111229 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:31.111347 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:31.111398 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:31.111671 (+ 273us) server_negotiation.cc:299] Negotiation successful
0924 04:12:31.111745 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 04:12:31.112131 20526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fd7c227ea4e4b1cba69109c7c5b6ddd" candidate_uuid: "b93c03ff9c7b43618ca45d7c398ecc01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19835f8931d54b468a37bf3b79546223" is_pre_election: true
I20260924 04:12:31.112493 20526 raft_consensus.cc:2454] T 0fd7c227ea4e4b1cba69109c7c5b6ddd P 19835f8931d54b468a37bf3b79546223 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b93c03ff9c7b43618ca45d7c398ecc01 in term 0.
I20260924 04:12:31.112573 20264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fd7c227ea4e4b1cba69109c7c5b6ddd" candidate_uuid: "b93c03ff9c7b43618ca45d7c398ecc01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d2142cdc3cf4b6eb6b4def03be15d93" is_pre_election: true
I20260924 04:12:31.112900 20264 raft_consensus.cc:2454] T 0fd7c227ea4e4b1cba69109c7c5b6ddd P 8d2142cdc3cf4b6eb6b4def03be15d93 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b93c03ff9c7b43618ca45d7c398ecc01 in term 0.
I20260924 04:12:31.113065 20331 leader_election.cc:302] T 0fd7c227ea4e4b1cba69109c7c5b6ddd P b93c03ff9c7b43618ca45d7c398ecc01 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19835f8931d54b468a37bf3b79546223, b93c03ff9c7b43618ca45d7c398ecc01; no voters:
I20260924 04:12:31.113384 20597 raft_consensus.cc:2781] T 0fd7c227ea4e4b1cba69109c7c5b6ddd P b93c03ff9c7b43618ca45d7c398ecc01 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:12:31.113479 20597 raft_consensus.cc:482] T 0fd7c227ea4e4b1cba69109c7c5b6ddd P b93c03ff9c7b43618ca45d7c398ecc01 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:12:31.113559 20597 raft_consensus.cc:3037] T 0fd7c227ea4e4b1cba69109c7c5b6ddd P b93c03ff9c7b43618ca45d7c398ecc01 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:12:31.114562 20597 raft_consensus.cc:504] T 0fd7c227ea4e4b1cba69109c7c5b6ddd P b93c03ff9c7b43618ca45d7c398ecc01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b93c03ff9c7b43618ca45d7c398ecc01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37051 } } peers { permanent_uuid: "8d2142cdc3cf4b6eb6b4def03be15d93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32831 } } peers { permanent_uuid: "19835f8931d54b468a37bf3b79546223" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38103 } }
I20260924 04:12:31.114930 20597 leader_election.cc:288] T 0fd7c227ea4e4b1cba69109c7c5b6ddd P b93c03ff9c7b43618ca45d7c398ecc01 [CANDIDATE]: Term 1 election: Requested vote from peers 8d2142cdc3cf4b6eb6b4def03be15d93 (127.0.98.1:32831), 19835f8931d54b468a37bf3b79546223 (127.0.98.3:38103)
I20260924 04:12:31.115255 20526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fd7c227ea4e4b1cba69109c7c5b6ddd" candidate_uuid: "b93c03ff9c7b43618ca45d7c398ecc01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19835f8931d54b468a37bf3b79546223"
I20260924 04:12:31.115293 20264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fd7c227ea4e4b1cba69109c7c5b6ddd" candidate_uuid: "b93c03ff9c7b43618ca45d7c398ecc01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d2142cdc3cf4b6eb6b4def03be15d93"
I20260924 04:12:31.115361 20526 raft_consensus.cc:3037] T 0fd7c227ea4e4b1cba69109c7c5b6ddd P 19835f8931d54b468a37bf3b79546223 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:12:31.115418 20264 raft_consensus.cc:3037] T 0fd7c227ea4e4b1cba69109c7c5b6ddd P 8d2142cdc3cf4b6eb6b4def03be15d93 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:12:31.116529 20526 raft_consensus.cc:2454] T 0fd7c227ea4e4b1cba69109c7c5b6ddd P 19835f8931d54b468a37bf3b79546223 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b93c03ff9c7b43618ca45d7c398ecc01 in term 1.
I20260924 04:12:31.116771 20264 raft_consensus.cc:2454] T 0fd7c227ea4e4b1cba69109c7c5b6ddd P 8d2142cdc3cf4b6eb6b4def03be15d93 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b93c03ff9c7b43618ca45d7c398ecc01 in term 1.
I20260924 04:12:31.116852 20331 leader_election.cc:302] T 0fd7c227ea4e4b1cba69109c7c5b6ddd P b93c03ff9c7b43618ca45d7c398ecc01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19835f8931d54b468a37bf3b79546223, b93c03ff9c7b43618ca45d7c398ecc01; no voters:
I20260924 04:12:31.117038 20597 raft_consensus.cc:2781] T 0fd7c227ea4e4b1cba69109c7c5b6ddd P b93c03ff9c7b43618ca45d7c398ecc01 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:12:31.117295 20597 raft_consensus.cc:686] T 0fd7c227ea4e4b1cba69109c7c5b6ddd P b93c03ff9c7b43618ca45d7c398ecc01 [term 1 LEADER]: Becoming Leader. State: Replica: b93c03ff9c7b43618ca45d7c398ecc01, State: Running, Role: LEADER
I20260924 04:12:31.117774 20597 consensus_queue.cc:237] T 0fd7c227ea4e4b1cba69109c7c5b6ddd P b93c03ff9c7b43618ca45d7c398ecc01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b93c03ff9c7b43618ca45d7c398ecc01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37051 } } peers { permanent_uuid: "8d2142cdc3cf4b6eb6b4def03be15d93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32831 } } peers { permanent_uuid: "19835f8931d54b468a37bf3b79546223" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38103 } }
W20260924 04:12:31.119683 20311 tablet.cc:2369] T 0fd7c227ea4e4b1cba69109c7c5b6ddd P 8d2142cdc3cf4b6eb6b4def03be15d93: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:12:31.121120 20121 catalog_manager.cc:5495] T 0fd7c227ea4e4b1cba69109c7c5b6ddd P b93c03ff9c7b43618ca45d7c398ecc01 reported cstate change: term changed from 0 to 1, leader changed from <none> to b93c03ff9c7b43618ca45d7c398ecc01 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "b93c03ff9c7b43618ca45d7c398ecc01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b93c03ff9c7b43618ca45d7c398ecc01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d2142cdc3cf4b6eb6b4def03be15d93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19835f8931d54b468a37bf3b79546223" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38103 } health_report { overall_health: UNKNOWN } } }
I20260924 04:12:31.135710 20585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:31.132420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39340 (local address 127.0.98.2:37051)
0924 04:12:31.132546 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:31.132550 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:31.132562 (+ 12us) server_negotiation.cc:407] Connection header received
0924 04:12:31.132630 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:31.132632 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:31.132727 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:31.132825 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:12:31.133040 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:31.133537 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:31.134383 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:31.134581 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:31.134653 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:31.134906 (+ 253us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:12:31.135303 (+ 397us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:12:31.135392 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:31.135525 (+ 133us) server_negotiation.cc:299] Negotiation successful
0924 04:12:31.135577 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 04:12:31.141832 20526 raft_consensus.cc:1261] T 0fd7c227ea4e4b1cba69109c7c5b6ddd P 19835f8931d54b468a37bf3b79546223 [term 1 FOLLOWER]: Refusing update from remote peer b93c03ff9c7b43618ca45d7c398ecc01: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:12:31.141825 20264 raft_consensus.cc:1261] T 0fd7c227ea4e4b1cba69109c7c5b6ddd P 8d2142cdc3cf4b6eb6b4def03be15d93 [term 1 FOLLOWER]: Refusing update from remote peer b93c03ff9c7b43618ca45d7c398ecc01: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:12:31.142572 20599 consensus_queue.cc:1035] T 0fd7c227ea4e4b1cba69109c7c5b6ddd P b93c03ff9c7b43618ca45d7c398ecc01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d2142cdc3cf4b6eb6b4def03be15d93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:12:31.142791 20597 consensus_queue.cc:1035] T 0fd7c227ea4e4b1cba69109c7c5b6ddd P b93c03ff9c7b43618ca45d7c398ecc01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19835f8931d54b468a37bf3b79546223" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:12:31.150112 20604 mvcc.cc:205] Tried to move back new op lower bound from 7332754027068825600 to 7332754026981289984. Current Snapshot: MvccSnapshot[applied={T|T < 7332754027068825600}]
I20260924 04:12:31.151628 20603 mvcc.cc:205] Tried to move back new op lower bound from 7332754027068825600 to 7332754026981289984. Current Snapshot: MvccSnapshot[applied={T|T < 7332754027068825600}]
I20260924 04:12:31.158918 20121 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40948:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:12:31.159077 20121 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40948:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:12:31.161746 20121 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7eef76e1cdb645b8bd8996a6e879d01d: Sending DeleteTablet for 3 replicas of tablet 0fd7c227ea4e4b1cba69109c7c5b6ddd
I20260924 04:12:31.162619 20244 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fd7c227ea4e4b1cba69109c7c5b6ddd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:12:31 UTC) from {username='slave'} at 127.0.0.1:60122
I20260924 04:12:31.162622 20506 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fd7c227ea4e4b1cba69109c7c5b6ddd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:12:31 UTC) from {username='slave'} at 127.0.0.1:39120
I20260924 04:12:31.163216 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 20182
I20260924 04:12:31.163347 20612 tablet_replica.cc:330] T 0fd7c227ea4e4b1cba69109c7c5b6ddd P 19835f8931d54b468a37bf3b79546223: stopping tablet replica
I20260924 04:12:31.163306 20375 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fd7c227ea4e4b1cba69109c7c5b6ddd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:12:31 UTC) from {username='slave'} at 127.0.0.1:39334
I20260924 04:12:31.163678 20612 raft_consensus.cc:2229] T 0fd7c227ea4e4b1cba69109c7c5b6ddd P 19835f8931d54b468a37bf3b79546223 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:12:31.163935 20613 tablet_replica.cc:330] T 0fd7c227ea4e4b1cba69109c7c5b6ddd P b93c03ff9c7b43618ca45d7c398ecc01: stopping tablet replica
I20260924 04:12:31.164079 20612 raft_consensus.cc:2258] T 0fd7c227ea4e4b1cba69109c7c5b6ddd P 19835f8931d54b468a37bf3b79546223 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:12:31.164327 20613 raft_consensus.cc:2229] T 0fd7c227ea4e4b1cba69109c7c5b6ddd P b93c03ff9c7b43618ca45d7c398ecc01 [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:12:31.164788 20613 raft_consensus.cc:2258] T 0fd7c227ea4e4b1cba69109c7c5b6ddd P b93c03ff9c7b43618ca45d7c398ecc01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:12:31.166107 20612 ts_tablet_manager.cc:1905] T 0fd7c227ea4e4b1cba69109c7c5b6ddd P 19835f8931d54b468a37bf3b79546223: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:12:31.169590 20612 ts_tablet_manager.cc:1918] T 0fd7c227ea4e4b1cba69109c7c5b6ddd P 19835f8931d54b468a37bf3b79546223: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:12:31.169677 20613 ts_tablet_manager.cc:1905] T 0fd7c227ea4e4b1cba69109c7c5b6ddd P b93c03ff9c7b43618ca45d7c398ecc01: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:12:31.169694 20612 log.cc:1196] T 0fd7c227ea4e4b1cba69109c7c5b6ddd P 19835f8931d54b468a37bf3b79546223: Deleting WAL directory at /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222898807720-392-0/minicluster-data/ts-2/wal/wals/0fd7c227ea4e4b1cba69109c7c5b6ddd
I20260924 04:12:31.170024 20612 ts_tablet_manager.cc:1939] T 0fd7c227ea4e4b1cba69109c7c5b6ddd P 19835f8931d54b468a37bf3b79546223: Deleting consensus metadata
I20260924 04:12:31.171337 20109 catalog_manager.cc:4841] TS 19835f8931d54b468a37bf3b79546223 (127.0.98.3:38103): tablet 0fd7c227ea4e4b1cba69109c7c5b6ddd (table test-table [id=3f8618fa890b4170a8dfa6cafd7849df]) successfully deleted
W20260924 04:12:31.172211 20108 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:32831 (error 108)
I20260924 04:12:31.172288 20613 ts_tablet_manager.cc:1918] T 0fd7c227ea4e4b1cba69109c7c5b6ddd P b93c03ff9c7b43618ca45d7c398ecc01: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:12:31.172382 20613 log.cc:1196] T 0fd7c227ea4e4b1cba69109c7c5b6ddd P b93c03ff9c7b43618ca45d7c398ecc01: Deleting WAL directory at /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790222898807720-392-0/minicluster-data/ts-1/wal/wals/0fd7c227ea4e4b1cba69109c7c5b6ddd
I20260924 04:12:31.172667 20613 ts_tablet_manager.cc:1939] T 0fd7c227ea4e4b1cba69109c7c5b6ddd P b93c03ff9c7b43618ca45d7c398ecc01: Deleting consensus metadata
I20260924 04:12:31.173074 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 20313
I20260924 04:12:31.173344 20584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:31.173033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32831 (local address 127.0.0.1:60138)
0924 04:12:31.173162 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:31.173248 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:32831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260924 04:12:31.174139 20108 catalog_manager.cc:4584] TS 8d2142cdc3cf4b6eb6b4def03be15d93 (127.0.98.1:32831): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0fd7c227ea4e4b1cba69109c7c5b6ddd: Network error: Client connection negotiation failed: client connection to 127.0.98.1:32831: connect: Connection refused (error 111)
I20260924 04:12:31.178483 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 20444
I20260924 04:12:31.179351 20584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:31.178680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37051 (local address 127.0.0.1:39352)
0924 04:12:31.179191 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:31.179259 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:37051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419}
I20260924 04:12:31.183945 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 20091
2026-09-24T04:12:31Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1046 ms)
[----------] 6 tests from EncryptionPolicyTest (10160 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790222898807720-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:12:31 dist-test-slave-00ld krb5kdc[20617](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:12:31 dist-test-slave-00ld krb5kdc[20617](info): set up 2 sockets
Sep 24 04:12:31 dist-test-slave-00ld krb5kdc[20617](info): commencing operation
krb5kdc: starting...
W20260924 04:12:32.224730 392 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790222898807720-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790222898807720-392-0/krb5kdc/test-user.keytab.
Sep 24 04:12:32 dist-test-slave-00ld krb5kdc[20617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223152, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:12:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:12:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790222898807720-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790222898807720-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 04:12:32.344583 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790222898807720-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790222898807720-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790222898807720-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39599
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790222898807720-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39599
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:12:32.452700 20630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:12:32.453027 20630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:12:32.453114 20630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:12:32.457201 20630 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:12:32.457304 20630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:12:32.457343 20630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:12:32.457381 20630 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:12:32.457438 20630 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:12:32.462824 20630 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790222898807720-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790222898807720-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39599
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790222898807720-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39599
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20630
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790222898807720-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:12:32.463973 20630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:12:32.465030 20630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:12:32.473731 20636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:12:32.473754 20638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:12:32.473704 20635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:12:32.474056 20630 server_base.cc:1034] running on GCE node
I20260924 04:12:32.474596 20630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:12:32.475721 20630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:12:32.476876 20630 hybrid_clock.cc:648] HybridClock initialized: now 1790223152476837 us; error 48 us; skew 500 ppm
Sep 24 04:12:32 dist-test-slave-00ld krb5kdc[20617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223152, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:12:32.479596 20630 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 04:12:32.480504 20630 webserver.cc:460] Webserver started at http://127.0.0.1:43273/ using document root <none> and password file <none>
I20260924 04:12:32.481175 20630 fs_manager.cc:359] Metadata directory not provided
I20260924 04:12:32.481274 20630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:12:32.481504 20630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:12:32.483523 20630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790222898807720-392-0/minicluster-data/master-0/data/instance:
uuid: "b0c261e6f2b44007a70d4cd087eb934f"
format_stamp: "Formatted at 2026-09-24 04:12:32 on dist-test-slave-00ld"
server_key: "976c63f90fbb2b35f4db2d9c553a5887"
server_key_iv: "6eeb3ad4249a71a99a6f49d13c966394"
server_key_version: "encryptionkey@0"
I20260924 04:12:32.484098 20630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790222898807720-392-0/minicluster-data/master-0/wal/instance:
uuid: "b0c261e6f2b44007a70d4cd087eb934f"
format_stamp: "Formatted at 2026-09-24 04:12:32 on dist-test-slave-00ld"
server_key: "976c63f90fbb2b35f4db2d9c553a5887"
server_key_iv: "6eeb3ad4249a71a99a6f49d13c966394"
server_key_version: "encryptionkey@0"
I20260924 04:12:32.487890 20630 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 04:12:32.490615 20645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:12:32.491999 20630 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 04:12:32.492228 20630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790222898807720-392-0/minicluster-data/master-0/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790222898807720-392-0/minicluster-data/master-0/wal
uuid: "b0c261e6f2b44007a70d4cd087eb934f"
format_stamp: "Formatted at 2026-09-24 04:12:32 on dist-test-slave-00ld"
server_key: "976c63f90fbb2b35f4db2d9c553a5887"
server_key_iv: "6eeb3ad4249a71a99a6f49d13c966394"
server_key_version: "encryptionkey@0"
I20260924 04:12:32.492362 20630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790222898807720-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790222898807720-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790222898807720-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:12:32.515123 20630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:12:32.518272 20630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:12:32.518482 20630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:12:32.525458 20630 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39599
I20260924 04:12:32.525481 20697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39599 every 8 connection(s)
I20260924 04:12:32.526350 20630 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:12:32.529191 20698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:12:32.530809 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 20630
I20260924 04:12:32.530958 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790222898807720-392-0/minicluster-data/master-0/wal/instance
I20260924 04:12:32.531250 392 external_mini_cluster.cc:1428] Setting key bd4649d32591011fdef107b67f1072ad
I20260924 04:12:32.536281 20698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0c261e6f2b44007a70d4cd087eb934f: Bootstrap starting.
I20260924 04:12:32.539024 20698 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b0c261e6f2b44007a70d4cd087eb934f: Neither blocks nor log segments found. Creating new log.
I20260924 04:12:32.539960 20698 log.cc:824] T 00000000000000000000000000000000 P b0c261e6f2b44007a70d4cd087eb934f: Log is configured to *not* fsync() on all Append() calls
Sep 24 04:12:32 dist-test-slave-00ld krb5kdc[20617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223152, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 04:12:32.542374 20698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0c261e6f2b44007a70d4cd087eb934f: No bootstrap required, opened a new log
I20260924 04:12:32.547587 20698 raft_consensus.cc:348] T 00000000000000000000000000000000 P b0c261e6f2b44007a70d4cd087eb934f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0c261e6f2b44007a70d4cd087eb934f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39599 } }
I20260924 04:12:32.547844 20701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:32.532776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34991 (local address 127.0.0.1:39599)
0924 04:12:32.533183 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:32.533206 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:32.533250 (+ 44us) server_negotiation.cc:407] Connection header received
0924 04:12:32.534234 (+ 984us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:32.534259 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:32.534725 (+ 466us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:32.535118 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:12:32.535770 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:32.537377 (+ 1607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:32.538216 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:32.538962 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:32.539148 (+ 186us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:32.542254 (+ 3106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:12:32.542260 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:12:32.542277 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:12:32.542308 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:12:32.544373 (+ 2065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:12:32.544983 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:12:32.544992 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:12:32.545000 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:12:32.545184 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:12:32.545591 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:12:32.545594 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:12:32.545596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:12:32.546111 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:12:32.546358 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:32.546572 (+ 214us) server_negotiation.cc:299] Negotiation successful
0924 04:12:32.546831 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":99,"threads_started":1}
I20260924 04:12:32.547865 20698 raft_consensus.cc:374] T 00000000000000000000000000000000 P b0c261e6f2b44007a70d4cd087eb934f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:12:32.547916 20698 raft_consensus.cc:729] T 00000000000000000000000000000000 P b0c261e6f2b44007a70d4cd087eb934f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0c261e6f2b44007a70d4cd087eb934f, State: Initialized, Role: FOLLOWER
I20260924 04:12:32.548417 20698 consensus_queue.cc:260] T 00000000000000000000000000000000 P b0c261e6f2b44007a70d4cd087eb934f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0c261e6f2b44007a70d4cd087eb934f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39599 } }
I20260924 04:12:32.548557 20698 raft_consensus.cc:388] T 00000000000000000000000000000000 P b0c261e6f2b44007a70d4cd087eb934f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:12:32.548658 20698 raft_consensus.cc:482] T 00000000000000000000000000000000 P b0c261e6f2b44007a70d4cd087eb934f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:12:32.548810 20698 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b0c261e6f2b44007a70d4cd087eb934f [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:12:32.550009 20698 raft_consensus.cc:504] T 00000000000000000000000000000000 P b0c261e6f2b44007a70d4cd087eb934f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0c261e6f2b44007a70d4cd087eb934f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39599 } }
I20260924 04:12:32.550397 20698 leader_election.cc:302] T 00000000000000000000000000000000 P b0c261e6f2b44007a70d4cd087eb934f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b0c261e6f2b44007a70d4cd087eb934f; no voters:
I20260924 04:12:32.550781 20698 leader_election.cc:288] T 00000000000000000000000000000000 P b0c261e6f2b44007a70d4cd087eb934f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:12:32.550887 20703 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b0c261e6f2b44007a70d4cd087eb934f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:12:32.551159 20703 raft_consensus.cc:686] T 00000000000000000000000000000000 P b0c261e6f2b44007a70d4cd087eb934f [term 1 LEADER]: Becoming Leader. State: Replica: b0c261e6f2b44007a70d4cd087eb934f, State: Running, Role: LEADER
I20260924 04:12:32.551280 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:12:32.551452 20703 consensus_queue.cc:237] T 00000000000000000000000000000000 P b0c261e6f2b44007a70d4cd087eb934f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0c261e6f2b44007a70d4cd087eb934f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39599 } }
I20260924 04:12:32.552148 20698 sys_catalog.cc:564] T 00000000000000000000000000000000 P b0c261e6f2b44007a70d4cd087eb934f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:12:32.552573 20705 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0c261e6f2b44007a70d4cd087eb934f [sys.catalog]: SysCatalogTable state changed. Reason: New leader b0c261e6f2b44007a70d4cd087eb934f. Latest consensus state: current_term: 1 leader_uuid: "b0c261e6f2b44007a70d4cd087eb934f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0c261e6f2b44007a70d4cd087eb934f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39599 } } }
I20260924 04:12:32.552727 20705 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0c261e6f2b44007a70d4cd087eb934f [sys.catalog]: This master's current role is: LEADER
I20260924 04:12:32.552994 20704 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0c261e6f2b44007a70d4cd087eb934f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b0c261e6f2b44007a70d4cd087eb934f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0c261e6f2b44007a70d4cd087eb934f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39599 } } }
I20260924 04:12:32.553081 20704 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0c261e6f2b44007a70d4cd087eb934f [sys.catalog]: This master's current role is: LEADER
I20260924 04:12:32.553153 20711 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:12:32.556454 20711 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:12:32.562872 20711 catalog_manager.cc:1347] Generated new cluster ID: 52fe14eb08974a86a736327b946b1900
I20260924 04:12:32.562943 20711 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:12:32.564182 20701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:32.553729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48367 (local address 127.0.0.1:39599)
0924 04:12:32.553849 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:32.553856 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:32.554721 (+ 865us) server_negotiation.cc:407] Connection header received
0924 04:12:32.554927 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:32.554932 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:32.554982 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:32.555035 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:12:32.556042 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:32.556821 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:32.557967 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:32.558337 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:32.558453 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:32.560158 (+ 1705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:12:32.560165 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:12:32.560169 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:12:32.560194 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:12:32.562134 (+ 1940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:12:32.562729 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:12:32.562733 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:12:32.562734 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:12:32.562820 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:12:32.563214 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:12:32.563218 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:12:32.563219 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:12:32.563430 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:12:32.563576 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:32.563913 (+ 337us) server_negotiation.cc:299] Negotiation successful
0924 04:12:32.564014 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 04:12:32.579504 20711 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:12:32.580749 20711 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:12:32.593411 20711 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b0c261e6f2b44007a70d4cd087eb934f: Generated new TSK 0
I20260924 04:12:32.594120 20711 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:12:32.605825 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 20630
W20260924 04:12:32.611352 20615 connection.cc:535] client connection to 127.0.0.1:39599 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39599): Connection reset by peer (error 104)
2026-09-24T04:12:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1419 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790222898807720-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:12:32 dist-test-slave-00ld krb5kdc[20729](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:12:32 dist-test-slave-00ld krb5kdc[20729](info): set up 2 sockets
Sep 24 04:12:32 dist-test-slave-00ld krb5kdc[20729](info): commencing operation
krb5kdc: starting...
W20260924 04:12:33.644819 392 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790222898807720-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790222898807720-392-0/krb5kdc/test-user.keytab.
Sep 24 04:12:33 dist-test-slave-00ld krb5kdc[20729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223153, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:12:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:12:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790222898807720-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790222898807720-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 04:12:33.765880 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790222898807720-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790222898807720-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790222898807720-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35679
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790222898807720-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35679
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:12:33.877345 20742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:12:33.877599 20742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:12:33.877638 20742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:12:33.881527 20742 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:12:33.881590 20742 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:12:33.881609 20742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:12:33.881626 20742 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:12:33.881641 20742 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:12:33.887534 20742 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790222898807720-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790222898807720-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35679
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790222898807720-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20742
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790222898807720-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:12:33.888543 20742 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:12:33.889547 20742 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:12:33.898068 20750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:12:33.898167 20742 server_base.cc:1034] running on GCE node
W20260924 04:12:33.898054 20748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:12:33.898054 20747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:12:33.898813 20742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:12:33.899854 20742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:12:33.901011 20742 hybrid_clock.cc:648] HybridClock initialized: now 1790223153900982 us; error 48 us; skew 500 ppm
Sep 24 04:12:33 dist-test-slave-00ld krb5kdc[20729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223153, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:12:33.903836 20742 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 04:12:33.904831 20742 webserver.cc:460] Webserver started at http://127.0.0.1:37439/ using document root <none> and password file <none>
I20260924 04:12:33.905395 20742 fs_manager.cc:359] Metadata directory not provided
I20260924 04:12:33.905486 20742 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:12:33.905717 20742 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:12:33.907749 20742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790222898807720-392-0/minicluster-data/master-0/data/instance:
uuid: "357e102ca5dd4ff19fafb9aae739ab09"
format_stamp: "Formatted at 2026-09-24 04:12:33 on dist-test-slave-00ld"
server_key: "ff78e779b5616ea513aff830486f1a34"
server_key_iv: "c0c3c9e03f7f6edbed8a2ca4077bd5ae"
server_key_version: "encryptionkey@0"
I20260924 04:12:33.908321 20742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790222898807720-392-0/minicluster-data/master-0/wal/instance:
uuid: "357e102ca5dd4ff19fafb9aae739ab09"
format_stamp: "Formatted at 2026-09-24 04:12:33 on dist-test-slave-00ld"
server_key: "ff78e779b5616ea513aff830486f1a34"
server_key_iv: "c0c3c9e03f7f6edbed8a2ca4077bd5ae"
server_key_version: "encryptionkey@0"
I20260924 04:12:33.912132 20742 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260924 04:12:33.914630 20757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:12:33.915925 20742 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:12:33.916078 20742 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790222898807720-392-0/minicluster-data/master-0/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790222898807720-392-0/minicluster-data/master-0/wal
uuid: "357e102ca5dd4ff19fafb9aae739ab09"
format_stamp: "Formatted at 2026-09-24 04:12:33 on dist-test-slave-00ld"
server_key: "ff78e779b5616ea513aff830486f1a34"
server_key_iv: "c0c3c9e03f7f6edbed8a2ca4077bd5ae"
server_key_version: "encryptionkey@0"
I20260924 04:12:33.916217 20742 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790222898807720-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790222898807720-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790222898807720-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:12:33.936784 20742 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:12:33.939817 20742 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:12:33.940030 20742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:12:33.947211 20742 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35679
I20260924 04:12:33.947227 20809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35679 every 8 connection(s)
I20260924 04:12:33.948182 20742 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:12:33.950933 20810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:12:33.951886 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 20742
I20260924 04:12:33.952042 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790222898807720-392-0/minicluster-data/master-0/wal/instance
I20260924 04:12:33.952390 392 external_mini_cluster.cc:1428] Setting key d552cd539f4b448f3985d21a6245301e
I20260924 04:12:33.957110 20810 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 357e102ca5dd4ff19fafb9aae739ab09: Bootstrap starting.
I20260924 04:12:33.959461 20810 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 357e102ca5dd4ff19fafb9aae739ab09: Neither blocks nor log segments found. Creating new log.
I20260924 04:12:33.960366 20810 log.cc:824] T 00000000000000000000000000000000 P 357e102ca5dd4ff19fafb9aae739ab09: Log is configured to *not* fsync() on all Append() calls
I20260924 04:12:33.961757 20813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:33.953881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60341 (local address 127.0.0.1:35679)
0924 04:12:33.954331 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:33.954345 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:33.954380 (+ 35us) server_negotiation.cc:407] Connection header received
0924 04:12:33.955007 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:33.955022 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:33.955360 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:33.955695 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:12:33.956089 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:33.957112 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:33.958074 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:33.958550 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:33.958729 (+ 179us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:33.959766 (+ 1037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:12:33.959775 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:12:33.959794 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:12:33.960299 (+ 505us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:12:33.960363 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:12:33.960396 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:12:33.960536 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:33.960810 (+ 274us) server_negotiation.cc:299] Negotiation successful
0924 04:12:33.960956 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":137,"threads_started":1}
I20260924 04:12:33.962574 20810 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 357e102ca5dd4ff19fafb9aae739ab09: No bootstrap required, opened a new log
I20260924 04:12:33.964639 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:12:33.966341 20810 raft_consensus.cc:348] T 00000000000000000000000000000000 P 357e102ca5dd4ff19fafb9aae739ab09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "357e102ca5dd4ff19fafb9aae739ab09" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35679 } }
I20260924 04:12:33.966616 20810 raft_consensus.cc:374] T 00000000000000000000000000000000 P 357e102ca5dd4ff19fafb9aae739ab09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:12:33.966677 20810 raft_consensus.cc:729] T 00000000000000000000000000000000 P 357e102ca5dd4ff19fafb9aae739ab09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 357e102ca5dd4ff19fafb9aae739ab09, State: Initialized, Role: FOLLOWER
I20260924 04:12:33.967164 20810 consensus_queue.cc:260] T 00000000000000000000000000000000 P 357e102ca5dd4ff19fafb9aae739ab09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "357e102ca5dd4ff19fafb9aae739ab09" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35679 } }
I20260924 04:12:33.967350 20810 raft_consensus.cc:388] T 00000000000000000000000000000000 P 357e102ca5dd4ff19fafb9aae739ab09 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:12:33.967425 20810 raft_consensus.cc:482] T 00000000000000000000000000000000 P 357e102ca5dd4ff19fafb9aae739ab09 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:12:33.967533 20810 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 357e102ca5dd4ff19fafb9aae739ab09 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:12:33.968611 20810 raft_consensus.cc:504] T 00000000000000000000000000000000 P 357e102ca5dd4ff19fafb9aae739ab09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "357e102ca5dd4ff19fafb9aae739ab09" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35679 } }
I20260924 04:12:33.968972 20810 leader_election.cc:302] T 00000000000000000000000000000000 P 357e102ca5dd4ff19fafb9aae739ab09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 357e102ca5dd4ff19fafb9aae739ab09; no voters:
I20260924 04:12:33.969317 20810 leader_election.cc:288] T 00000000000000000000000000000000 P 357e102ca5dd4ff19fafb9aae739ab09 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:12:33.969389 20820 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 357e102ca5dd4ff19fafb9aae739ab09 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:12:33.969606 20820 raft_consensus.cc:686] T 00000000000000000000000000000000 P 357e102ca5dd4ff19fafb9aae739ab09 [term 1 LEADER]: Becoming Leader. State: Replica: 357e102ca5dd4ff19fafb9aae739ab09, State: Running, Role: LEADER
I20260924 04:12:33.969911 20820 consensus_queue.cc:237] T 00000000000000000000000000000000 P 357e102ca5dd4ff19fafb9aae739ab09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "357e102ca5dd4ff19fafb9aae739ab09" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35679 } }
I20260924 04:12:33.970619 20810 sys_catalog.cc:564] T 00000000000000000000000000000000 P 357e102ca5dd4ff19fafb9aae739ab09 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:12:33.971041 20822 sys_catalog.cc:455] T 00000000000000000000000000000000 P 357e102ca5dd4ff19fafb9aae739ab09 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "357e102ca5dd4ff19fafb9aae739ab09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "357e102ca5dd4ff19fafb9aae739ab09" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35679 } } }
I20260924 04:12:33.971212 20822 sys_catalog.cc:458] T 00000000000000000000000000000000 P 357e102ca5dd4ff19fafb9aae739ab09 [sys.catalog]: This master's current role is: LEADER
I20260924 04:12:33.971431 20821 sys_catalog.cc:455] T 00000000000000000000000000000000 P 357e102ca5dd4ff19fafb9aae739ab09 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 357e102ca5dd4ff19fafb9aae739ab09. Latest consensus state: current_term: 1 leader_uuid: "357e102ca5dd4ff19fafb9aae739ab09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "357e102ca5dd4ff19fafb9aae739ab09" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35679 } } }
I20260924 04:12:33.971552 20821 sys_catalog.cc:458] T 00000000000000000000000000000000 P 357e102ca5dd4ff19fafb9aae739ab09 [sys.catalog]: This master's current role is: LEADER
I20260924 04:12:33.972381 20813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:33.966644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52225 (local address 127.0.0.1:35679)
0924 04:12:33.966768 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:33.966772 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:33.967012 (+ 240us) server_negotiation.cc:407] Connection header received
0924 04:12:33.967109 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:33.967111 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:33.967147 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:33.967219 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:12:33.967549 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:33.968088 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:33.969258 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:33.969622 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:33.969748 (+ 126us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:33.971925 (+ 2177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:12:33.971929 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:12:33.971932 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:12:33.971944 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:12:33.971975 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:12:33.971985 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:12:33.972081 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:33.972206 (+ 125us) server_negotiation.cc:299] Negotiation successful
0924 04:12:33.972240 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260924 04:12:33.975147 20835 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 357e102ca5dd4ff19fafb9aae739ab09: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:12:33.975247 20835 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:12:33.975329 20825 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:12:33.976058 20825 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:12:33.981076 20825 catalog_manager.cc:1347] Generated new cluster ID: 4528062c88de4b55b776a60926e8dced
I20260924 04:12:33.981141 20825 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:12:34.000725 20825 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:12:34.002043 20825 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:12:34.019716 20825 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 357e102ca5dd4ff19fafb9aae739ab09: Generated new TSK 0
I20260924 04:12:34.020490 20825 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:12:34.026716 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 20742
W20260924 04:12:34.032387 20727 connection.cc:535] client connection to 127.0.0.1:35679 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35679): Connection reset by peer (error 104)
2026-09-24T04:12:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1421 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790222898807720-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:12:34 dist-test-slave-00ld krb5kdc[20841](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:12:34 dist-test-slave-00ld krb5kdc[20841](info): set up 2 sockets
Sep 24 04:12:34 dist-test-slave-00ld krb5kdc[20841](info): commencing operation
krb5kdc: starting...
W20260924 04:12:35.068913 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790222898807720-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790222898807720-392-0/krb5kdc/test-user.keytab.
Sep 24 04:12:35 dist-test-slave-00ld krb5kdc[20841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223155, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:12:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:12:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790222898807720-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790222898807720-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 04:12:35.189401 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790222898807720-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790222898807720-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790222898807720-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37697
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790222898807720-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37697
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:12:35.298416 20854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:12:35.298728 20854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:12:35.298815 20854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:12:35.302780 20854 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:12:35.302862 20854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:12:35.302902 20854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:12:35.302958 20854 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:12:35.302996 20854 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:12:35.308238 20854 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790222898807720-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790222898807720-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37697
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790222898807720-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20854
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790222898807720-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:12:35.309296 20854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:12:35.310292 20854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:12:35.318672 20860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:12:35.318817 20862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:12:35.318820 20859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:12:35.318960 20854 server_base.cc:1034] running on GCE node
I20260924 04:12:35.319643 20854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:12:35.320762 20854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:12:35.321945 20854 hybrid_clock.cc:648] HybridClock initialized: now 1790223155321933 us; error 33 us; skew 500 ppm
Sep 24 04:12:35 dist-test-slave-00ld krb5kdc[20841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223155, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:12:35.324575 20854 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 04:12:35.325553 20854 webserver.cc:460] Webserver started at http://127.0.0.1:40521/ using document root <none> and password file <none>
I20260924 04:12:35.326112 20854 fs_manager.cc:359] Metadata directory not provided
I20260924 04:12:35.326172 20854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:12:35.326390 20854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:12:35.328531 20854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790222898807720-392-0/minicluster-data/master-0/data/instance:
uuid: "2fd952566ff049afad337356dfe4aeef"
format_stamp: "Formatted at 2026-09-24 04:12:35 on dist-test-slave-00ld"
server_key: "48a3fed71c5332e19a347d7717bffca4"
server_key_iv: "0ac847570857a6f39266a20c86fa7a26"
server_key_version: "encryptionkey@0"
I20260924 04:12:35.329164 20854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790222898807720-392-0/minicluster-data/master-0/wal/instance:
uuid: "2fd952566ff049afad337356dfe4aeef"
format_stamp: "Formatted at 2026-09-24 04:12:35 on dist-test-slave-00ld"
server_key: "48a3fed71c5332e19a347d7717bffca4"
server_key_iv: "0ac847570857a6f39266a20c86fa7a26"
server_key_version: "encryptionkey@0"
I20260924 04:12:35.332773 20854 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:12:35.335095 20869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:12:35.336262 20854 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:12:35.336390 20854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790222898807720-392-0/minicluster-data/master-0/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790222898807720-392-0/minicluster-data/master-0/wal
uuid: "2fd952566ff049afad337356dfe4aeef"
format_stamp: "Formatted at 2026-09-24 04:12:35 on dist-test-slave-00ld"
server_key: "48a3fed71c5332e19a347d7717bffca4"
server_key_iv: "0ac847570857a6f39266a20c86fa7a26"
server_key_version: "encryptionkey@0"
I20260924 04:12:35.336486 20854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790222898807720-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790222898807720-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790222898807720-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:12:35.342725 20854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:12:35.351049 20854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:12:35.351214 20854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:12:35.358062 20854 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37697
I20260924 04:12:35.358137 20921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37697 every 8 connection(s)
I20260924 04:12:35.358982 20854 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:12:35.361877 20922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:12:35.365494 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 20854
I20260924 04:12:35.365640 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790222898807720-392-0/minicluster-data/master-0/wal/instance
I20260924 04:12:35.365926 392 external_mini_cluster.cc:1428] Setting key 6289d4fd367918cbb01e575d3d95d68e
I20260924 04:12:35.368261 20922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fd952566ff049afad337356dfe4aeef: Bootstrap starting.
I20260924 04:12:35.370820 20922 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2fd952566ff049afad337356dfe4aeef: Neither blocks nor log segments found. Creating new log.
I20260924 04:12:35.371555 20922 log.cc:824] T 00000000000000000000000000000000 P 2fd952566ff049afad337356dfe4aeef: Log is configured to *not* fsync() on all Append() calls
I20260924 04:12:35.371565 20925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:35.367525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47169 (local address 127.0.0.1:37697)
0924 04:12:35.368083 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:35.368131 (+ 48us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:35.368178 (+ 47us) server_negotiation.cc:407] Connection header received
0924 04:12:35.368960 (+ 782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:35.368979 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:35.369271 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:35.369547 (+ 276us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:12:35.370357 (+ 810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:12:35.370361 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:12:35.370373 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:12:35.370383 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:12:35.370433 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:12:35.370452 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:12:35.370535 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:35.370660 (+ 125us) server_negotiation.cc:299] Negotiation successful
0924 04:12:35.370840 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":143,"threads_started":1}
I20260924 04:12:35.373641 20922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fd952566ff049afad337356dfe4aeef: No bootstrap required, opened a new log
I20260924 04:12:35.374456 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:12:35.378218 20925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:35.376388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54025 (local address 127.0.0.1:37697)
0924 04:12:35.376506 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:35.376509 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:35.376784 (+ 275us) server_negotiation.cc:407] Connection header received
0924 04:12:35.376960 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:35.376963 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:35.376989 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:35.377051 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:12:35.377814 (+ 763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:12:35.377817 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:12:35.377819 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:12:35.377829 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:12:35.377852 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:12:35.377862 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:12:35.377929 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:35.378046 (+ 117us) server_negotiation.cc:299] Negotiation successful
0924 04:12:35.378074 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 04:12:35.378222 20922 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2fd952566ff049afad337356dfe4aeef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fd952566ff049afad337356dfe4aeef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37697 } }
I20260924 04:12:35.378607 20922 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2fd952566ff049afad337356dfe4aeef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:12:35.378670 20922 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2fd952566ff049afad337356dfe4aeef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fd952566ff049afad337356dfe4aeef, State: Initialized, Role: FOLLOWER
I20260924 04:12:35.379187 20922 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2fd952566ff049afad337356dfe4aeef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fd952566ff049afad337356dfe4aeef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37697 } }
I20260924 04:12:35.379375 20922 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2fd952566ff049afad337356dfe4aeef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:12:35.379446 20922 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2fd952566ff049afad337356dfe4aeef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:12:35.379549 20922 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2fd952566ff049afad337356dfe4aeef [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:12:35.380465 20922 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2fd952566ff049afad337356dfe4aeef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fd952566ff049afad337356dfe4aeef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37697 } }
I20260924 04:12:35.380878 20922 leader_election.cc:302] T 00000000000000000000000000000000 P 2fd952566ff049afad337356dfe4aeef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2fd952566ff049afad337356dfe4aeef; no voters:
I20260924 04:12:35.381181 20922 leader_election.cc:288] T 00000000000000000000000000000000 P 2fd952566ff049afad337356dfe4aeef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:12:35.381300 20932 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2fd952566ff049afad337356dfe4aeef [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:12:35.381537 20932 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2fd952566ff049afad337356dfe4aeef [term 1 LEADER]: Becoming Leader. State: Replica: 2fd952566ff049afad337356dfe4aeef, State: Running, Role: LEADER
I20260924 04:12:35.381870 20932 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2fd952566ff049afad337356dfe4aeef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fd952566ff049afad337356dfe4aeef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37697 } }
I20260924 04:12:35.382400 20922 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2fd952566ff049afad337356dfe4aeef [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:12:35.383603 20933 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fd952566ff049afad337356dfe4aeef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2fd952566ff049afad337356dfe4aeef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fd952566ff049afad337356dfe4aeef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37697 } } }
I20260924 04:12:35.383603 20934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fd952566ff049afad337356dfe4aeef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2fd952566ff049afad337356dfe4aeef. Latest consensus state: current_term: 1 leader_uuid: "2fd952566ff049afad337356dfe4aeef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fd952566ff049afad337356dfe4aeef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37697 } } }
I20260924 04:12:35.383795 20933 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fd952566ff049afad337356dfe4aeef [sys.catalog]: This master's current role is: LEADER
I20260924 04:12:35.383795 20934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fd952566ff049afad337356dfe4aeef [sys.catalog]: This master's current role is: LEADER
I20260924 04:12:35.384169 20941 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:12:35.387084 20941 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:12:35.392519 20941 catalog_manager.cc:1347] Generated new cluster ID: 2bb6e76c996349e9a47abc45209566a3
I20260924 04:12:35.392589 20941 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:12:35.408205 20941 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:12:35.408593 20941 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:12:35.426419 20941 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2fd952566ff049afad337356dfe4aeef: Generated new TSK 0
I20260924 04:12:35.427311 20941 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:12:35.435609 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 20854
2026-09-24T04:12:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1408 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790222898807720-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:12:35 dist-test-slave-00ld krb5kdc[20953](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:12:35 dist-test-slave-00ld krb5kdc[20953](info): set up 2 sockets
Sep 24 04:12:35 dist-test-slave-00ld krb5kdc[20953](info): commencing operation
krb5kdc: starting...
W20260924 04:12:36.472886 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790222898807720-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790222898807720-392-0/krb5kdc/test-user.keytab.
Sep 24 04:12:36 dist-test-slave-00ld krb5kdc[20953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:12:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:12:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790222898807720-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790222898807720-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 04:12:36.591410 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790222898807720-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790222898807720-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790222898807720-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43797
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790222898807720-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43797
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:12:36.702193 20966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:12:36.702503 20966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:12:36.702586 20966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:12:36.706818 20966 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:12:36.706900 20966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:12:36.706938 20966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:12:36.706995 20966 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:12:36.707032 20966 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:12:36.712457 20966 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790222898807720-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790222898807720-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43797
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790222898807720-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43797
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20966
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790222898807720-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:12:36.713603 20966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:12:36.714655 20966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:12:36.723335 20971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:12:36.723325 20974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:12:36.723326 20972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:12:36.723381 20966 server_base.cc:1034] running on GCE node
I20260924 04:12:36.724166 20966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:12:36.725353 20966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:12:36.726536 20966 hybrid_clock.cc:648] HybridClock initialized: now 1790223156726510 us; error 40 us; skew 500 ppm
Sep 24 04:12:36 dist-test-slave-00ld krb5kdc[20953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223156, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:12:36.736658 20966 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 04:12:36.737708 20966 webserver.cc:460] Webserver started at http://127.0.0.1:35027/ using document root <none> and password file <none>
I20260924 04:12:36.738307 20966 fs_manager.cc:359] Metadata directory not provided
I20260924 04:12:36.738396 20966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:12:36.738633 20966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:12:36.740803 20966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790222898807720-392-0/minicluster-data/master-0/data/instance:
uuid: "7fdd883f38e84c0ebc3d8f2537b6c2f6"
format_stamp: "Formatted at 2026-09-24 04:12:36 on dist-test-slave-00ld"
server_key: "ecf115c11bd7c8cceb1c64a3537163d9"
server_key_iv: "d49992304965515e8f2a1ae715b1b170"
server_key_version: "encryptionkey@0"
I20260924 04:12:36.741375 20966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790222898807720-392-0/minicluster-data/master-0/wal/instance:
uuid: "7fdd883f38e84c0ebc3d8f2537b6c2f6"
format_stamp: "Formatted at 2026-09-24 04:12:36 on dist-test-slave-00ld"
server_key: "ecf115c11bd7c8cceb1c64a3537163d9"
server_key_iv: "d49992304965515e8f2a1ae715b1b170"
server_key_version: "encryptionkey@0"
I20260924 04:12:36.745040 20966 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:12:36.747452 20981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:12:36.748657 20966 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:12:36.748852 20966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790222898807720-392-0/minicluster-data/master-0/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790222898807720-392-0/minicluster-data/master-0/wal
uuid: "7fdd883f38e84c0ebc3d8f2537b6c2f6"
format_stamp: "Formatted at 2026-09-24 04:12:36 on dist-test-slave-00ld"
server_key: "ecf115c11bd7c8cceb1c64a3537163d9"
server_key_iv: "d49992304965515e8f2a1ae715b1b170"
server_key_version: "encryptionkey@0"
I20260924 04:12:36.748966 20966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790222898807720-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790222898807720-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790222898807720-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:12:36.761943 20966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:12:36.764945 20966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:12:36.765151 20966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:12:36.772349 20966 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43797
I20260924 04:12:36.772362 21033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43797 every 8 connection(s)
I20260924 04:12:36.773342 20966 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:12:36.776176 21034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:12:36.777205 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 20966
I20260924 04:12:36.777379 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790222898807720-392-0/minicluster-data/master-0/wal/instance
I20260924 04:12:36.777652 392 external_mini_cluster.cc:1428] Setting key c6db3feb31fde2e6c1364e89795b49f3
I20260924 04:12:36.782160 21034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fdd883f38e84c0ebc3d8f2537b6c2f6: Bootstrap starting.
I20260924 04:12:36.784308 21034 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7fdd883f38e84c0ebc3d8f2537b6c2f6: Neither blocks nor log segments found. Creating new log.
I20260924 04:12:36.785091 21034 log.cc:824] T 00000000000000000000000000000000 P 7fdd883f38e84c0ebc3d8f2537b6c2f6: Log is configured to *not* fsync() on all Append() calls
Sep 24 04:12:36 dist-test-slave-00ld krb5kdc[20953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 04:12:36.787096 21034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fdd883f38e84c0ebc3d8f2537b6c2f6: No bootstrap required, opened a new log
I20260924 04:12:36.791848 21037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:36.779049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44063 (local address 127.0.0.1:43797)
0924 04:12:36.779464 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:36.779482 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:36.779517 (+ 35us) server_negotiation.cc:407] Connection header received
0924 04:12:36.780121 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:36.780137 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:36.780476 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:36.780841 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:12:36.781655 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:36.782654 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:36.783504 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:36.783970 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:36.784133 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:36.786698 (+ 2565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:12:36.786705 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:12:36.786721 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:12:36.786751 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:12:36.788795 (+ 2044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:12:36.789230 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:12:36.789236 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:12:36.789241 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:12:36.789399 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:12:36.789680 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:12:36.789685 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:12:36.789687 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:12:36.790193 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:12:36.790391 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:36.790603 (+ 212us) server_negotiation.cc:299] Negotiation successful
0924 04:12:36.790906 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":130,"threads_started":1}
I20260924 04:12:36.791810 21034 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7fdd883f38e84c0ebc3d8f2537b6c2f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fdd883f38e84c0ebc3d8f2537b6c2f6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43797 } }
I20260924 04:12:36.792081 21034 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7fdd883f38e84c0ebc3d8f2537b6c2f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:12:36.792140 21034 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7fdd883f38e84c0ebc3d8f2537b6c2f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fdd883f38e84c0ebc3d8f2537b6c2f6, State: Initialized, Role: FOLLOWER
I20260924 04:12:36.792577 21034 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7fdd883f38e84c0ebc3d8f2537b6c2f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fdd883f38e84c0ebc3d8f2537b6c2f6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43797 } }
I20260924 04:12:36.792750 21034 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7fdd883f38e84c0ebc3d8f2537b6c2f6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:12:36.792824 21034 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7fdd883f38e84c0ebc3d8f2537b6c2f6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:12:36.792922 21034 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7fdd883f38e84c0ebc3d8f2537b6c2f6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:12:36.793889 21034 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7fdd883f38e84c0ebc3d8f2537b6c2f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fdd883f38e84c0ebc3d8f2537b6c2f6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43797 } }
I20260924 04:12:36.794255 21034 leader_election.cc:302] T 00000000000000000000000000000000 P 7fdd883f38e84c0ebc3d8f2537b6c2f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7fdd883f38e84c0ebc3d8f2537b6c2f6; no voters:
I20260924 04:12:36.794610 21034 leader_election.cc:288] T 00000000000000000000000000000000 P 7fdd883f38e84c0ebc3d8f2537b6c2f6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:12:36.794710 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:12:36.794718 21039 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7fdd883f38e84c0ebc3d8f2537b6c2f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:12:36.794960 21039 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7fdd883f38e84c0ebc3d8f2537b6c2f6 [term 1 LEADER]: Becoming Leader. State: Replica: 7fdd883f38e84c0ebc3d8f2537b6c2f6, State: Running, Role: LEADER
I20260924 04:12:36.795305 21039 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7fdd883f38e84c0ebc3d8f2537b6c2f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fdd883f38e84c0ebc3d8f2537b6c2f6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43797 } }
I20260924 04:12:36.795928 21034 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7fdd883f38e84c0ebc3d8f2537b6c2f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:12:36.796957 21041 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fdd883f38e84c0ebc3d8f2537b6c2f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7fdd883f38e84c0ebc3d8f2537b6c2f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fdd883f38e84c0ebc3d8f2537b6c2f6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43797 } } }
I20260924 04:12:36.797082 21041 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fdd883f38e84c0ebc3d8f2537b6c2f6 [sys.catalog]: This master's current role is: LEADER
I20260924 04:12:36.797148 21040 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fdd883f38e84c0ebc3d8f2537b6c2f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7fdd883f38e84c0ebc3d8f2537b6c2f6. Latest consensus state: current_term: 1 leader_uuid: "7fdd883f38e84c0ebc3d8f2537b6c2f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fdd883f38e84c0ebc3d8f2537b6c2f6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43797 } } }
I20260924 04:12:36.797263 21040 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fdd883f38e84c0ebc3d8f2537b6c2f6 [sys.catalog]: This master's current role is: LEADER
I20260924 04:12:36.797541 21053 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:12:36.800774 21053 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:12:36.806648 21037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:36.796858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39839 (local address 127.0.0.1:43797)
0924 04:12:36.796977 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:36.796982 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:36.798234 (+ 1252us) server_negotiation.cc:407] Connection header received
0924 04:12:36.798274 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:36.798277 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:36.798318 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:36.799186 (+ 868us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:12:36.799280 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:36.800002 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:36.801126 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:36.801449 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:36.801897 (+ 448us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:36.803139 (+ 1242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:12:36.803143 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:12:36.803145 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:12:36.803168 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:12:36.804594 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:12:36.805031 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:12:36.805036 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:12:36.805037 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:12:36.805118 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:12:36.805419 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:12:36.805423 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:12:36.805424 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:12:36.805615 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:12:36.805728 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:36.806398 (+ 670us) server_negotiation.cc:299] Negotiation successful
0924 04:12:36.806505 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 04:12:36.806746 21053 catalog_manager.cc:1347] Generated new cluster ID: 0a3caeea5700463b9391c33afdefd625
I20260924 04:12:36.806946 21053 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:12:36.816290 21053 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:12:36.817234 21053 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:12:36.821671 21053 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7fdd883f38e84c0ebc3d8f2537b6c2f6: Generated new TSK 0
I20260924 04:12:36.822309 21053 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:12:36.826848 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 20966
2026-09-24T04:12:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1390 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790222898807720-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:12:36 dist-test-slave-00ld krb5kdc[21065](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:12:36 dist-test-slave-00ld krb5kdc[21065](info): set up 2 sockets
Sep 24 04:12:36 dist-test-slave-00ld krb5kdc[21065](info): commencing operation
krb5kdc: starting...
W20260924 04:12:37.881021 392 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790222898807720-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790222898807720-392-0/krb5kdc/test-user.keytab.
Sep 24 04:12:37 dist-test-slave-00ld krb5kdc[21065](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223157, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:12:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:12:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790222898807720-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790222898807720-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 04:12:38.000938 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790222898807720-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790222898807720-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790222898807720-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38913
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790222898807720-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38913
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:12:38.106374 21078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:12:38.106674 21078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:12:38.106758 21078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:12:38.110918 21078 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:12:38.111004 21078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:12:38.111037 21078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:12:38.111089 21078 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:12:38.111131 21078 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:12:38.116262 21078 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790222898807720-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790222898807720-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38913
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790222898807720-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38913
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21078
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790222898807720-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:12:38.117313 21078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:12:38.118296 21078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:12:38.126573 21083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:12:38.126736 21078 server_base.cc:1034] running on GCE node
W20260924 04:12:38.126588 21084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:12:38.126585 21086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:12:38.127410 21078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:12:38.128481 21078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:12:38.129671 21078 hybrid_clock.cc:648] HybridClock initialized: now 1790223158129650 us; error 33 us; skew 500 ppm
Sep 24 04:12:38 dist-test-slave-00ld krb5kdc[21065](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223158, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:12:38.132546 21078 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 04:12:38.133451 21078 webserver.cc:460] Webserver started at http://127.0.0.1:44063/ using document root <none> and password file <none>
I20260924 04:12:38.134016 21078 fs_manager.cc:359] Metadata directory not provided
I20260924 04:12:38.134109 21078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:12:38.134338 21078 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:12:38.136374 21078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790222898807720-392-0/minicluster-data/master-0/data/instance:
uuid: "c2a0a4ab5ef749e6a6b5cdbf5a415bbc"
format_stamp: "Formatted at 2026-09-24 04:12:38 on dist-test-slave-00ld"
server_key: "f53a785a8c0f47fb22d59b24fc55fb37"
server_key_iv: "3c5dea7612c0352b162b467d8b86ab51"
server_key_version: "encryptionkey@0"
I20260924 04:12:38.136960 21078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790222898807720-392-0/minicluster-data/master-0/wal/instance:
uuid: "c2a0a4ab5ef749e6a6b5cdbf5a415bbc"
format_stamp: "Formatted at 2026-09-24 04:12:38 on dist-test-slave-00ld"
server_key: "f53a785a8c0f47fb22d59b24fc55fb37"
server_key_iv: "3c5dea7612c0352b162b467d8b86ab51"
server_key_version: "encryptionkey@0"
I20260924 04:12:38.140523 21078 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260924 04:12:38.142963 21093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:12:38.144189 21078 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:12:38.144341 21078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790222898807720-392-0/minicluster-data/master-0/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790222898807720-392-0/minicluster-data/master-0/wal
uuid: "c2a0a4ab5ef749e6a6b5cdbf5a415bbc"
format_stamp: "Formatted at 2026-09-24 04:12:38 on dist-test-slave-00ld"
server_key: "f53a785a8c0f47fb22d59b24fc55fb37"
server_key_iv: "3c5dea7612c0352b162b467d8b86ab51"
server_key_version: "encryptionkey@0"
I20260924 04:12:38.144456 21078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790222898807720-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790222898807720-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790222898807720-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:12:38.163833 21078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:12:38.167119 21078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:12:38.167325 21078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:12:38.174588 21078 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38913
I20260924 04:12:38.174616 21145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38913 every 8 connection(s)
I20260924 04:12:38.175522 21078 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:12:38.176644 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 21078
I20260924 04:12:38.176839 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790222898807720-392-0/minicluster-data/master-0/wal/instance
I20260924 04:12:38.177109 392 external_mini_cluster.cc:1428] Setting key df105270a6256dd108ffb10ed67fd11d
I20260924 04:12:38.178956 21146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:12:38.185951 21146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2a0a4ab5ef749e6a6b5cdbf5a415bbc: Bootstrap starting.
I20260924 04:12:38.187127 21149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:38.178486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39821 (local address 127.0.0.1:38913)
0924 04:12:38.178902 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:38.178922 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:38.178968 (+ 46us) server_negotiation.cc:407] Connection header received
0924 04:12:38.179793 (+ 825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:38.179815 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:38.180241 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:38.180617 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:12:38.181032 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:38.182537 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:38.183536 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:38.184213 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:38.184370 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:38.185229 (+ 859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:12:38.185236 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:12:38.185251 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:12:38.185738 (+ 487us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:12:38.185806 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:12:38.185836 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:12:38.185927 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:38.186036 (+ 109us) server_negotiation.cc:299] Negotiation successful
0924 04:12:38.186233 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":116,"threads_started":1}
I20260924 04:12:38.188369 21146 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c2a0a4ab5ef749e6a6b5cdbf5a415bbc: Neither blocks nor log segments found. Creating new log.
I20260924 04:12:38.189158 21146 log.cc:824] T 00000000000000000000000000000000 P c2a0a4ab5ef749e6a6b5cdbf5a415bbc: Log is configured to *not* fsync() on all Append() calls
I20260924 04:12:38.189936 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:12:38.191443 21146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2a0a4ab5ef749e6a6b5cdbf5a415bbc: No bootstrap required, opened a new log
I20260924 04:12:38.196106 21149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:38.192094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32921 (local address 127.0.0.1:38913)
0924 04:12:38.192236 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:38.192240 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:38.192431 (+ 191us) server_negotiation.cc:407] Connection header received
0924 04:12:38.192547 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:38.192549 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:38.192583 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:38.192646 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:12:38.192939 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:38.193498 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:38.194679 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:38.194890 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:38.194962 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:38.195715 (+ 753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:12:38.195719 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:12:38.195722 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:12:38.195731 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:12:38.195754 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:12:38.195763 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:12:38.195818 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:38.195927 (+ 109us) server_negotiation.cc:299] Negotiation successful
0924 04:12:38.195966 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 04:12:38.196038 21146 raft_consensus.cc:348] T 00000000000000000000000000000000 P c2a0a4ab5ef749e6a6b5cdbf5a415bbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2a0a4ab5ef749e6a6b5cdbf5a415bbc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38913 } }
I20260924 04:12:38.196337 21146 raft_consensus.cc:374] T 00000000000000000000000000000000 P c2a0a4ab5ef749e6a6b5cdbf5a415bbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:12:38.196403 21146 raft_consensus.cc:729] T 00000000000000000000000000000000 P c2a0a4ab5ef749e6a6b5cdbf5a415bbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2a0a4ab5ef749e6a6b5cdbf5a415bbc, State: Initialized, Role: FOLLOWER
I20260924 04:12:38.196882 21146 consensus_queue.cc:260] T 00000000000000000000000000000000 P c2a0a4ab5ef749e6a6b5cdbf5a415bbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2a0a4ab5ef749e6a6b5cdbf5a415bbc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38913 } }
I20260924 04:12:38.197057 21146 raft_consensus.cc:388] T 00000000000000000000000000000000 P c2a0a4ab5ef749e6a6b5cdbf5a415bbc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:12:38.197137 21146 raft_consensus.cc:482] T 00000000000000000000000000000000 P c2a0a4ab5ef749e6a6b5cdbf5a415bbc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:12:38.197237 21146 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c2a0a4ab5ef749e6a6b5cdbf5a415bbc [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:12:38.198024 21146 raft_consensus.cc:504] T 00000000000000000000000000000000 P c2a0a4ab5ef749e6a6b5cdbf5a415bbc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2a0a4ab5ef749e6a6b5cdbf5a415bbc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38913 } }
I20260924 04:12:38.198299 21146 leader_election.cc:302] T 00000000000000000000000000000000 P c2a0a4ab5ef749e6a6b5cdbf5a415bbc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c2a0a4ab5ef749e6a6b5cdbf5a415bbc; no voters:
I20260924 04:12:38.198573 21146 leader_election.cc:288] T 00000000000000000000000000000000 P c2a0a4ab5ef749e6a6b5cdbf5a415bbc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:12:38.198642 21156 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c2a0a4ab5ef749e6a6b5cdbf5a415bbc [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:12:38.198848 21156 raft_consensus.cc:686] T 00000000000000000000000000000000 P c2a0a4ab5ef749e6a6b5cdbf5a415bbc [term 1 LEADER]: Becoming Leader. State: Replica: c2a0a4ab5ef749e6a6b5cdbf5a415bbc, State: Running, Role: LEADER
I20260924 04:12:38.199198 21156 consensus_queue.cc:237] T 00000000000000000000000000000000 P c2a0a4ab5ef749e6a6b5cdbf5a415bbc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2a0a4ab5ef749e6a6b5cdbf5a415bbc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38913 } }
I20260924 04:12:38.199813 21146 sys_catalog.cc:564] T 00000000000000000000000000000000 P c2a0a4ab5ef749e6a6b5cdbf5a415bbc [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:12:38.200384 21156 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2a0a4ab5ef749e6a6b5cdbf5a415bbc [sys.catalog]: SysCatalogTable state changed. Reason: New leader c2a0a4ab5ef749e6a6b5cdbf5a415bbc. Latest consensus state: current_term: 1 leader_uuid: "c2a0a4ab5ef749e6a6b5cdbf5a415bbc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2a0a4ab5ef749e6a6b5cdbf5a415bbc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38913 } } }
I20260924 04:12:38.200531 21156 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2a0a4ab5ef749e6a6b5cdbf5a415bbc [sys.catalog]: This master's current role is: LEADER
I20260924 04:12:38.200875 21157 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2a0a4ab5ef749e6a6b5cdbf5a415bbc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c2a0a4ab5ef749e6a6b5cdbf5a415bbc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2a0a4ab5ef749e6a6b5cdbf5a415bbc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38913 } } }
I20260924 04:12:38.201023 21157 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2a0a4ab5ef749e6a6b5cdbf5a415bbc [sys.catalog]: This master's current role is: LEADER
W20260924 04:12:38.204286 21171 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c2a0a4ab5ef749e6a6b5cdbf5a415bbc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:12:38.204349 21171 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:12:38.204427 21163 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:12:38.205189 21163 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:12:38.210898 21163 catalog_manager.cc:1347] Generated new cluster ID: c4fa189bed5e45d3ab91a8a21f2d4427
I20260924 04:12:38.210976 21163 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:12:38.231294 21163 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:12:38.232184 21163 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:12:38.239559 21163 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c2a0a4ab5ef749e6a6b5cdbf5a415bbc: Generated new TSK 0
I20260924 04:12:38.240226 21163 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:12:38.247823 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 21078
2026-09-24T04:12:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1421 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790222898807720-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:12:38 dist-test-slave-00ld krb5kdc[21177](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:12:38 dist-test-slave-00ld krb5kdc[21177](info): set up 2 sockets
Sep 24 04:12:38 dist-test-slave-00ld krb5kdc[21177](info): commencing operation
krb5kdc: starting...
W20260924 04:12:39.369050 392 mini_kdc.cc:121] Time spent starting KDC: real 1.102s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790222898807720-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790222898807720-392-0/krb5kdc/test-user.keytab.
Sep 24 04:12:39 dist-test-slave-00ld krb5kdc[21177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:12:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:12:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790222898807720-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790222898807720-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 04:12:39.488065 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790222898807720-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790222898807720-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790222898807720-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33539
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790222898807720-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33539
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:12:39.597025 21190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:12:39.597288 21190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:12:39.597330 21190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:12:39.601279 21190 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:12:39.601333 21190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:12:39.601352 21190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:12:39.601368 21190 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:12:39.601383 21190 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:12:39.606637 21190 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790222898807720-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790222898807720-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33539
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790222898807720-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21190
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790222898807720-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:12:39.607589 21190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:12:39.608542 21190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:12:39.616786 21198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:12:39.616796 21195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:12:39.616848 21196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:12:39.616899 21190 server_base.cc:1034] running on GCE node
I20260924 04:12:39.617581 21190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:12:39.618706 21190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:12:39.619874 21190 hybrid_clock.cc:648] HybridClock initialized: now 1790223159619856 us; error 30 us; skew 500 ppm
Sep 24 04:12:39 dist-test-slave-00ld krb5kdc[21177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223159, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:12:39.622529 21190 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 04:12:39.623315 21190 webserver.cc:460] Webserver started at http://127.0.0.1:36115/ using document root <none> and password file <none>
I20260924 04:12:39.623817 21190 fs_manager.cc:359] Metadata directory not provided
I20260924 04:12:39.623872 21190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:12:39.624027 21190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:12:39.625998 21190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790222898807720-392-0/minicluster-data/master-0/data/instance:
uuid: "2133c280d65f46e086914cf8e7598010"
format_stamp: "Formatted at 2026-09-24 04:12:39 on dist-test-slave-00ld"
server_key: "57aacc1e1fe62e7b199dd3c5a2ca4223"
server_key_iv: "59d25258f77a93b631c5d729fcfc4703"
server_key_version: "encryptionkey@0"
I20260924 04:12:39.626487 21190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790222898807720-392-0/minicluster-data/master-0/wal/instance:
uuid: "2133c280d65f46e086914cf8e7598010"
format_stamp: "Formatted at 2026-09-24 04:12:39 on dist-test-slave-00ld"
server_key: "57aacc1e1fe62e7b199dd3c5a2ca4223"
server_key_iv: "59d25258f77a93b631c5d729fcfc4703"
server_key_version: "encryptionkey@0"
I20260924 04:12:39.629977 21190 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:12:39.632239 21205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:12:39.633474 21190 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:12:39.633613 21190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790222898807720-392-0/minicluster-data/master-0/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790222898807720-392-0/minicluster-data/master-0/wal
uuid: "2133c280d65f46e086914cf8e7598010"
format_stamp: "Formatted at 2026-09-24 04:12:39 on dist-test-slave-00ld"
server_key: "57aacc1e1fe62e7b199dd3c5a2ca4223"
server_key_iv: "59d25258f77a93b631c5d729fcfc4703"
server_key_version: "encryptionkey@0"
I20260924 04:12:39.633734 21190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790222898807720-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790222898807720-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790222898807720-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:12:39.638285 21190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:12:39.641076 21190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:12:39.641273 21190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:12:39.647989 21190 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33539
I20260924 04:12:39.648020 21257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33539 every 8 connection(s)
I20260924 04:12:39.648928 21190 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:12:39.651540 21258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:12:39.653822 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 21190
I20260924 04:12:39.653941 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790222898807720-392-0/minicluster-data/master-0/wal/instance
I20260924 04:12:39.654217 392 external_mini_cluster.cc:1428] Setting key 7d80e63435cc045133b7f9ef88e06809
I20260924 04:12:39.658062 21258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2133c280d65f46e086914cf8e7598010: Bootstrap starting.
I20260924 04:12:39.659960 21261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:39.655664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33671 (local address 127.0.0.1:33539)
0924 04:12:39.656120 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:39.656140 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:39.656185 (+ 45us) server_negotiation.cc:407] Connection header received
0924 04:12:39.657051 (+ 866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:39.657078 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:39.657440 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:39.657820 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:12:39.658561 (+ 741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:12:39.658566 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:12:39.658581 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:12:39.658592 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:12:39.658653 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:12:39.658690 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:12:39.658823 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:39.658890 (+ 67us) server_negotiation.cc:299] Negotiation successful
0924 04:12:39.659036 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":115,"threads_started":1}
I20260924 04:12:39.660877 21258 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2133c280d65f46e086914cf8e7598010: Neither blocks nor log segments found. Creating new log.
I20260924 04:12:39.661624 21258 log.cc:824] T 00000000000000000000000000000000 P 2133c280d65f46e086914cf8e7598010: Log is configured to *not* fsync() on all Append() calls
I20260924 04:12:39.662772 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:12:39.663589 21258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2133c280d65f46e086914cf8e7598010: No bootstrap required, opened a new log
I20260924 04:12:39.667018 21261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:39.664924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44761 (local address 127.0.0.1:33539)
0924 04:12:39.665062 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:39.665067 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:39.665706 (+ 639us) server_negotiation.cc:407] Connection header received
0924 04:12:39.665835 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:39.665837 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:39.665866 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:39.665931 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:12:39.666613 (+ 682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:12:39.666615 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:12:39.666617 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:12:39.666627 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:12:39.666650 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:12:39.666660 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:12:39.666738 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:39.666848 (+ 110us) server_negotiation.cc:299] Negotiation successful
0924 04:12:39.666883 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 04:12:39.668452 21258 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2133c280d65f46e086914cf8e7598010 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2133c280d65f46e086914cf8e7598010" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33539 } }
I20260924 04:12:39.668730 21258 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2133c280d65f46e086914cf8e7598010 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:12:39.668800 21258 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2133c280d65f46e086914cf8e7598010 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2133c280d65f46e086914cf8e7598010, State: Initialized, Role: FOLLOWER
I20260924 04:12:39.669236 21258 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2133c280d65f46e086914cf8e7598010 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2133c280d65f46e086914cf8e7598010" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33539 } }
I20260924 04:12:39.669394 21258 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2133c280d65f46e086914cf8e7598010 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:12:39.669497 21258 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2133c280d65f46e086914cf8e7598010 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:12:39.669607 21258 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2133c280d65f46e086914cf8e7598010 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:12:39.670575 21258 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2133c280d65f46e086914cf8e7598010 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2133c280d65f46e086914cf8e7598010" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33539 } }
I20260924 04:12:39.670962 21258 leader_election.cc:302] T 00000000000000000000000000000000 P 2133c280d65f46e086914cf8e7598010 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2133c280d65f46e086914cf8e7598010; no voters:
I20260924 04:12:39.671300 21258 leader_election.cc:288] T 00000000000000000000000000000000 P 2133c280d65f46e086914cf8e7598010 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:12:39.671416 21268 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2133c280d65f46e086914cf8e7598010 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:12:39.671687 21268 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2133c280d65f46e086914cf8e7598010 [term 1 LEADER]: Becoming Leader. State: Replica: 2133c280d65f46e086914cf8e7598010, State: Running, Role: LEADER
I20260924 04:12:39.672057 21268 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2133c280d65f46e086914cf8e7598010 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2133c280d65f46e086914cf8e7598010" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33539 } }
I20260924 04:12:39.672636 21258 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2133c280d65f46e086914cf8e7598010 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:12:39.673679 21270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2133c280d65f46e086914cf8e7598010 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2133c280d65f46e086914cf8e7598010. Latest consensus state: current_term: 1 leader_uuid: "2133c280d65f46e086914cf8e7598010" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2133c280d65f46e086914cf8e7598010" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33539 } } }
I20260924 04:12:39.673816 21270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2133c280d65f46e086914cf8e7598010 [sys.catalog]: This master's current role is: LEADER
I20260924 04:12:39.673903 21269 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2133c280d65f46e086914cf8e7598010 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2133c280d65f46e086914cf8e7598010" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2133c280d65f46e086914cf8e7598010" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33539 } } }
I20260924 04:12:39.673985 21269 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2133c280d65f46e086914cf8e7598010 [sys.catalog]: This master's current role is: LEADER
I20260924 04:12:39.674234 21277 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:12:39.677546 21277 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:12:39.683187 21277 catalog_manager.cc:1347] Generated new cluster ID: f48d3f5c72f94445bd263bce14c3e7bf
I20260924 04:12:39.683275 21277 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:12:39.692735 21277 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:12:39.693053 21277 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:12:39.702378 21277 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2133c280d65f46e086914cf8e7598010: Generated new TSK 0
I20260924 04:12:39.703229 21277 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:12:39.708844 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 21190
2026-09-24T04:12:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1460 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790222898807720-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:12:39 dist-test-slave-00ld krb5kdc[21289](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:12:39 dist-test-slave-00ld krb5kdc[21289](info): set up 2 sockets
Sep 24 04:12:39 dist-test-slave-00ld krb5kdc[21289](info): commencing operation
krb5kdc: starting...
W20260924 04:12:40.741046 392 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790222898807720-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790222898807720-392-0/krb5kdc/test-user.keytab.
Sep 24 04:12:40 dist-test-slave-00ld krb5kdc[21289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223160, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:12:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:12:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790222898807720-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790222898807720-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 04:12:40.861950 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790222898807720-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790222898807720-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790222898807720-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41291
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790222898807720-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41291
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:12:40.986539 21302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:12:40.986958 21302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:12:40.987054 21302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:12:40.992769 21302 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:12:40.992903 21302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:12:40.992959 21302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:12:40.993007 21302 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:12:40.993054 21302 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:12:41.001240 21302 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790222898807720-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790222898807720-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41291
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790222898807720-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41291
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21302
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790222898807720-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:12:41.002348 21302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:12:41.003330 21302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:12:41.011857 21308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:12:41.011938 21302 server_base.cc:1034] running on GCE node
W20260924 04:12:41.011983 21310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:12:41.011900 21307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:12:41.012660 21302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:12:41.013814 21302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:12:41.014978 21302 hybrid_clock.cc:648] HybridClock initialized: now 1790223161014951 us; error 39 us; skew 500 ppm
Sep 24 04:12:41 dist-test-slave-00ld krb5kdc[21289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223161, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:12:41.017887 21302 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 04:12:41.018705 21302 webserver.cc:460] Webserver started at http://127.0.0.1:33951/ using document root <none> and password file <none>
I20260924 04:12:41.019224 21302 fs_manager.cc:359] Metadata directory not provided
I20260924 04:12:41.019284 21302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:12:41.019441 21302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:12:41.021478 21302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790222898807720-392-0/minicluster-data/master-0/data/instance:
uuid: "1ac47e6c91e34d77bc11cb85655e10c1"
format_stamp: "Formatted at 2026-09-24 04:12:41 on dist-test-slave-00ld"
server_key: "a62f571aec3900309fbf5d52e575de6f"
server_key_iv: "ed0c18544e6940bcb7586e0e5cf23c41"
server_key_version: "encryptionkey@0"
I20260924 04:12:41.021976 21302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790222898807720-392-0/minicluster-data/master-0/wal/instance:
uuid: "1ac47e6c91e34d77bc11cb85655e10c1"
format_stamp: "Formatted at 2026-09-24 04:12:41 on dist-test-slave-00ld"
server_key: "a62f571aec3900309fbf5d52e575de6f"
server_key_iv: "ed0c18544e6940bcb7586e0e5cf23c41"
server_key_version: "encryptionkey@0"
I20260924 04:12:41.025550 21302 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:12:41.027802 21317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:12:41.029109 21302 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:12:41.029233 21302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790222898807720-392-0/minicluster-data/master-0/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790222898807720-392-0/minicluster-data/master-0/wal
uuid: "1ac47e6c91e34d77bc11cb85655e10c1"
format_stamp: "Formatted at 2026-09-24 04:12:41 on dist-test-slave-00ld"
server_key: "a62f571aec3900309fbf5d52e575de6f"
server_key_iv: "ed0c18544e6940bcb7586e0e5cf23c41"
server_key_version: "encryptionkey@0"
I20260924 04:12:41.029320 21302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790222898807720-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790222898807720-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790222898807720-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:12:41.053526 21302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:12:41.056875 21302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:12:41.057032 21302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:12:41.063884 21302 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41291
I20260924 04:12:41.063898 21369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41291 every 8 connection(s)
I20260924 04:12:41.064901 21302 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:12:41.067653 21370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:12:41.068244 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 21302
I20260924 04:12:41.068387 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790222898807720-392-0/minicluster-data/master-0/wal/instance
I20260924 04:12:41.068702 392 external_mini_cluster.cc:1428] Setting key 8c057d30c6132a1ab5957778cf5ff445
I20260924 04:12:41.074807 21370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ac47e6c91e34d77bc11cb85655e10c1: Bootstrap starting.
I20260924 04:12:41.077590 21370 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ac47e6c91e34d77bc11cb85655e10c1: Neither blocks nor log segments found. Creating new log.
Sep 24 04:12:41 dist-test-slave-00ld krb5kdc[21289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223160, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 04:12:41.078366 21370 log.cc:824] T 00000000000000000000000000000000 P 1ac47e6c91e34d77bc11cb85655e10c1: Log is configured to *not* fsync() on all Append() calls
I20260924 04:12:41.080770 21370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ac47e6c91e34d77bc11cb85655e10c1: No bootstrap required, opened a new log
I20260924 04:12:41.083864 21373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:41.070131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.47:45129 (local address 127.0.0.1:41291)
0924 04:12:41.070582 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:41.070604 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:41.070639 (+ 35us) server_negotiation.cc:407] Connection header received
0924 04:12:41.071293 (+ 654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:41.071310 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:41.071644 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:41.071927 (+ 283us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:12:41.072870 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:41.073911 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:41.075069 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:41.075541 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:41.075682 (+ 141us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:41.078535 (+ 2853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:12:41.078541 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:12:41.078555 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:12:41.078583 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:12:41.080572 (+ 1989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:12:41.081202 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:12:41.081208 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:12:41.081215 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:12:41.081349 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:12:41.081661 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:12:41.081664 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:12:41.081666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:12:41.082147 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:12:41.082361 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:41.082577 (+ 216us) server_negotiation.cc:299] Negotiation successful
0924 04:12:41.082826 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":155,"threads_started":1}
I20260924 04:12:41.085369 21370 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ac47e6c91e34d77bc11cb85655e10c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ac47e6c91e34d77bc11cb85655e10c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41291 } }
I20260924 04:12:41.085588 21370 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ac47e6c91e34d77bc11cb85655e10c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:12:41.085659 21370 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ac47e6c91e34d77bc11cb85655e10c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ac47e6c91e34d77bc11cb85655e10c1, State: Initialized, Role: FOLLOWER
I20260924 04:12:41.086105 21370 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ac47e6c91e34d77bc11cb85655e10c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ac47e6c91e34d77bc11cb85655e10c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41291 } }
I20260924 04:12:41.086288 21370 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ac47e6c91e34d77bc11cb85655e10c1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:12:41.086417 21370 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ac47e6c91e34d77bc11cb85655e10c1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:12:41.086525 21370 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ac47e6c91e34d77bc11cb85655e10c1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:12:41.087078 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:12:41.087618 21370 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ac47e6c91e34d77bc11cb85655e10c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ac47e6c91e34d77bc11cb85655e10c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41291 } }
I20260924 04:12:41.087963 21370 leader_election.cc:302] T 00000000000000000000000000000000 P 1ac47e6c91e34d77bc11cb85655e10c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ac47e6c91e34d77bc11cb85655e10c1; no voters:
I20260924 04:12:41.088333 21370 leader_election.cc:288] T 00000000000000000000000000000000 P 1ac47e6c91e34d77bc11cb85655e10c1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:12:41.088398 21378 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ac47e6c91e34d77bc11cb85655e10c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:12:41.088614 21378 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ac47e6c91e34d77bc11cb85655e10c1 [term 1 LEADER]: Becoming Leader. State: Replica: 1ac47e6c91e34d77bc11cb85655e10c1, State: Running, Role: LEADER
I20260924 04:12:41.088982 21378 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ac47e6c91e34d77bc11cb85655e10c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ac47e6c91e34d77bc11cb85655e10c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41291 } }
I20260924 04:12:41.089672 21370 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ac47e6c91e34d77bc11cb85655e10c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:12:41.090220 21378 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ac47e6c91e34d77bc11cb85655e10c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ac47e6c91e34d77bc11cb85655e10c1. Latest consensus state: current_term: 1 leader_uuid: "1ac47e6c91e34d77bc11cb85655e10c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ac47e6c91e34d77bc11cb85655e10c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41291 } } }
I20260924 04:12:41.090337 21378 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ac47e6c91e34d77bc11cb85655e10c1 [sys.catalog]: This master's current role is: LEADER
I20260924 04:12:41.090647 21380 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ac47e6c91e34d77bc11cb85655e10c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ac47e6c91e34d77bc11cb85655e10c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ac47e6c91e34d77bc11cb85655e10c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41291 } } }
I20260924 04:12:41.090759 21380 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ac47e6c91e34d77bc11cb85655e10c1 [sys.catalog]: This master's current role is: LEADER
W20260924 04:12:41.094605 21395 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1ac47e6c91e34d77bc11cb85655e10c1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:12:41.094728 21395 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:12:41.094818 21384 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:12:41.095762 21384 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:12:41.098340 21373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:41.089404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.47:37045 (local address 127.0.0.1:41291)
0924 04:12:41.089513 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:41.089517 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:41.089528 (+ 11us) server_negotiation.cc:407] Connection header received
0924 04:12:41.089568 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:41.089570 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:41.089605 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:41.089688 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:12:41.090582 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:41.091360 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:41.092483 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:41.092818 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:41.092924 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:41.094652 (+ 1728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:12:41.094656 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:12:41.094659 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:12:41.094687 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:12:41.096363 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:12:41.096972 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:12:41.096975 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:12:41.096977 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:12:41.097060 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:12:41.097428 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:12:41.097431 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:12:41.097432 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:12:41.097634 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:12:41.097782 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:41.098066 (+ 284us) server_negotiation.cc:299] Negotiation successful
0924 04:12:41.098166 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 04:12:41.101470 21384 catalog_manager.cc:1347] Generated new cluster ID: 07e166beff16477db5798fc3ce1de982
I20260924 04:12:41.101529 21384 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:12:41.107151 21384 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:12:41.108253 21384 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:12:41.124370 21384 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ac47e6c91e34d77bc11cb85655e10c1: Generated new TSK 0
I20260924 04:12:41.125200 21384 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:12:41.128502 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 21302
W20260924 04:12:41.133869 21287 connection.cc:535] client connection to 127.0.0.1:41291 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41291): Connection reset by peer (error 104)
2026-09-24T04:12:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1419 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790222898807720-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:12:41 dist-test-slave-00ld krb5kdc[21401](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:12:41 dist-test-slave-00ld krb5kdc[21401](info): set up 2 sockets
Sep 24 04:12:41 dist-test-slave-00ld krb5kdc[21401](info): commencing operation
krb5kdc: starting...
W20260924 04:12:42.156971 392 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790222898807720-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790222898807720-392-0/krb5kdc/test-user.keytab.
Sep 24 04:12:42 dist-test-slave-00ld krb5kdc[21401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:12:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:12:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790222898807720-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790222898807720-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 04:12:42.279037 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790222898807720-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790222898807720-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790222898807720-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42401
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790222898807720-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42401
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:12:42.386649 21414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:12:42.386978 21414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:12:42.387054 21414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:12:42.391405 21414 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:12:42.391491 21414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:12:42.391530 21414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:12:42.391566 21414 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:12:42.391601 21414 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:12:42.396843 21414 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790222898807720-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790222898807720-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42401
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790222898807720-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42401
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21414
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790222898807720-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:12:42.397913 21414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:12:42.398955 21414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:12:42.407284 21414 server_base.cc:1034] running on GCE node
W20260924 04:12:42.407234 21419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:12:42.407269 21420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:12:42.407234 21422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:12:42.408079 21414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:12:42.409135 21414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:12:42.410292 21414 hybrid_clock.cc:648] HybridClock initialized: now 1790223162410262 us; error 48 us; skew 500 ppm
Sep 24 04:12:42 dist-test-slave-00ld krb5kdc[21401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223162, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:12:42.419974 21414 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 04:12:42.420913 21414 webserver.cc:460] Webserver started at http://127.0.0.1:34105/ using document root <none> and password file <none>
I20260924 04:12:42.421442 21414 fs_manager.cc:359] Metadata directory not provided
I20260924 04:12:42.421501 21414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:12:42.421669 21414 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:12:42.423631 21414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790222898807720-392-0/minicluster-data/master-0/data/instance:
uuid: "8bb3158bffb941438d4fd41e806bf813"
format_stamp: "Formatted at 2026-09-24 04:12:42 on dist-test-slave-00ld"
server_key: "77bebcac1541d0671520a1c1ac1502ab"
server_key_iv: "4a329018e33b91a436a59ed44a4c892e"
server_key_version: "encryptionkey@0"
I20260924 04:12:42.424163 21414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790222898807720-392-0/minicluster-data/master-0/wal/instance:
uuid: "8bb3158bffb941438d4fd41e806bf813"
format_stamp: "Formatted at 2026-09-24 04:12:42 on dist-test-slave-00ld"
server_key: "77bebcac1541d0671520a1c1ac1502ab"
server_key_iv: "4a329018e33b91a436a59ed44a4c892e"
server_key_version: "encryptionkey@0"
I20260924 04:12:42.427690 21414 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 04:12:42.430044 21429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:12:42.431238 21414 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:12:42.431365 21414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790222898807720-392-0/minicluster-data/master-0/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790222898807720-392-0/minicluster-data/master-0/wal
uuid: "8bb3158bffb941438d4fd41e806bf813"
format_stamp: "Formatted at 2026-09-24 04:12:42 on dist-test-slave-00ld"
server_key: "77bebcac1541d0671520a1c1ac1502ab"
server_key_iv: "4a329018e33b91a436a59ed44a4c892e"
server_key_version: "encryptionkey@0"
I20260924 04:12:42.431468 21414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790222898807720-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790222898807720-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790222898807720-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:12:42.444594 21414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:12:42.447348 21414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:12:42.447554 21414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:12:42.454631 21414 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42401
I20260924 04:12:42.454658 21481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42401 every 8 connection(s)
I20260924 04:12:42.455587 21414 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:12:42.458374 21482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:12:42.463647 21482 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bb3158bffb941438d4fd41e806bf813: Bootstrap starting.
I20260924 04:12:42.464830 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 21414
I20260924 04:12:42.464988 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790222898807720-392-0/minicluster-data/master-0/wal/instance
I20260924 04:12:42.465277 392 external_mini_cluster.cc:1428] Setting key 5d9496863f6bfa4d3f0a8beb863f2881
I20260924 04:12:42.466507 21482 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8bb3158bffb941438d4fd41e806bf813: Neither blocks nor log segments found. Creating new log.
I20260924 04:12:42.467257 21482 log.cc:824] T 00000000000000000000000000000000 P 8bb3158bffb941438d4fd41e806bf813: Log is configured to *not* fsync() on all Append() calls
I20260924 04:12:42.469304 21482 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bb3158bffb941438d4fd41e806bf813: No bootstrap required, opened a new log
I20260924 04:12:42.473546 21482 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8bb3158bffb941438d4fd41e806bf813 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bb3158bffb941438d4fd41e806bf813" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42401 } }
I20260924 04:12:42.473789 21482 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8bb3158bffb941438d4fd41e806bf813 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:12:42.473868 21482 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8bb3158bffb941438d4fd41e806bf813 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bb3158bffb941438d4fd41e806bf813, State: Initialized, Role: FOLLOWER
I20260924 04:12:42.474063 21485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:42.466349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.47:57397 (local address 127.0.0.1:42401)
0924 04:12:42.466748 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:42.466764 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:42.466797 (+ 33us) server_negotiation.cc:407] Connection header received
0924 04:12:42.467457 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:42.467475 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:42.467895 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:42.468222 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:12:42.468529 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:42.469587 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:42.470793 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:42.471244 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:42.471358 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:42.472248 (+ 890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:12:42.472253 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:12:42.472265 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:12:42.472630 (+ 365us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:12:42.472737 (+ 107us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:12:42.472774 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:12:42.472875 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:42.473101 (+ 226us) server_negotiation.cc:299] Negotiation successful
0924 04:12:42.473277 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":128,"threads_started":1}
I20260924 04:12:42.474352 21482 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8bb3158bffb941438d4fd41e806bf813 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bb3158bffb941438d4fd41e806bf813" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42401 } }
I20260924 04:12:42.474509 21482 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8bb3158bffb941438d4fd41e806bf813 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:12:42.474572 21482 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8bb3158bffb941438d4fd41e806bf813 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:12:42.474663 21482 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8bb3158bffb941438d4fd41e806bf813 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:12:42.475574 21482 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8bb3158bffb941438d4fd41e806bf813 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bb3158bffb941438d4fd41e806bf813" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42401 } }
I20260924 04:12:42.475953 21482 leader_election.cc:302] T 00000000000000000000000000000000 P 8bb3158bffb941438d4fd41e806bf813 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8bb3158bffb941438d4fd41e806bf813; no voters:
I20260924 04:12:42.476279 21482 leader_election.cc:288] T 00000000000000000000000000000000 P 8bb3158bffb941438d4fd41e806bf813 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:12:42.476382 21487 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8bb3158bffb941438d4fd41e806bf813 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:12:42.476588 21487 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8bb3158bffb941438d4fd41e806bf813 [term 1 LEADER]: Becoming Leader. State: Replica: 8bb3158bffb941438d4fd41e806bf813, State: Running, Role: LEADER
I20260924 04:12:42.476948 21487 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8bb3158bffb941438d4fd41e806bf813 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bb3158bffb941438d4fd41e806bf813" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42401 } }
I20260924 04:12:42.477035 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:12:42.477489 21482 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8bb3158bffb941438d4fd41e806bf813 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:12:42.478559 21488 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bb3158bffb941438d4fd41e806bf813 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8bb3158bffb941438d4fd41e806bf813" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bb3158bffb941438d4fd41e806bf813" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42401 } } }
I20260924 04:12:42.478675 21488 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bb3158bffb941438d4fd41e806bf813 [sys.catalog]: This master's current role is: LEADER
I20260924 04:12:42.479007 21489 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bb3158bffb941438d4fd41e806bf813 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8bb3158bffb941438d4fd41e806bf813. Latest consensus state: current_term: 1 leader_uuid: "8bb3158bffb941438d4fd41e806bf813" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bb3158bffb941438d4fd41e806bf813" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42401 } } }
I20260924 04:12:42.479100 21500 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:12:42.479168 21489 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bb3158bffb941438d4fd41e806bf813 [sys.catalog]: This master's current role is: LEADER
I20260924 04:12:42.482232 21500 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:12:42.484999 21485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:42.479606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.47:36145 (local address 127.0.0.1:42401)
0924 04:12:42.479710 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:42.479714 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:42.480214 (+ 500us) server_negotiation.cc:407] Connection header received
0924 04:12:42.480371 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:42.480375 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:42.480419 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:42.480484 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:12:42.480869 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:42.481641 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:42.482900 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:42.483252 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:42.483349 (+ 97us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:42.484367 (+ 1018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:12:42.484370 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:12:42.484373 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:12:42.484385 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:12:42.484412 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:12:42.484425 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:12:42.484503 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:42.484841 (+ 338us) server_negotiation.cc:299] Negotiation successful
0924 04:12:42.484873 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 04:12:42.492558 21500 catalog_manager.cc:1347] Generated new cluster ID: 39949dc08429452c9c1dc14014b11b6f
I20260924 04:12:42.492630 21500 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:12:42.523547 21500 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:12:42.524531 21500 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:12:42.537818 21500 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8bb3158bffb941438d4fd41e806bf813: Generated new TSK 0
I20260924 04:12:42.538542 21500 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:12:42.551260 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 21414
W20260924 04:12:42.556190 21399 connection.cc:535] client connection to 127.0.0.1:42401 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42401): Connection reset by peer (error 104)
2026-09-24T04:12:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1422 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790222898807720-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:12:42 dist-test-slave-00ld krb5kdc[21513](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:12:42 dist-test-slave-00ld krb5kdc[21513](info): set up 2 sockets
Sep 24 04:12:42 dist-test-slave-00ld krb5kdc[21513](info): commencing operation
krb5kdc: starting...
W20260924 04:12:43.613063 392 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790222898807720-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790222898807720-392-0/krb5kdc/test-user.keytab.
Sep 24 04:12:43 dist-test-slave-00ld krb5kdc[21513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:12:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:12:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790222898807720-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790222898807720-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 04:12:43.733930 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790222898807720-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790222898807720-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790222898807720-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42007
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790222898807720-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42007
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:12:43.846474 21526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:12:43.846771 21526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:12:43.846843 21526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:12:43.850838 21526 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:12:43.850922 21526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:12:43.850960 21526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:12:43.851022 21526 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:12:43.851061 21526 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:12:43.856274 21526 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790222898807720-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790222898807720-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42007
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790222898807720-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42007
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21526
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790222898807720-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:12:43.857364 21526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:12:43.858364 21526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:12:43.866923 21531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:12:43.866927 21534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:12:43.866954 21532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:12:43.867084 21526 server_base.cc:1034] running on GCE node
I20260924 04:12:43.867887 21526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:12:43.869035 21526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:12:43.870237 21526 hybrid_clock.cc:648] HybridClock initialized: now 1790223163870201 us; error 47 us; skew 500 ppm
Sep 24 04:12:43 dist-test-slave-00ld krb5kdc[21513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223163, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:12:43.872962 21526 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 04:12:43.873816 21526 webserver.cc:460] Webserver started at http://127.0.0.1:46177/ using document root <none> and password file <none>
I20260924 04:12:43.874368 21526 fs_manager.cc:359] Metadata directory not provided
I20260924 04:12:43.874459 21526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:12:43.874692 21526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:12:43.876892 21526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790222898807720-392-0/minicluster-data/master-0/data/instance:
uuid: "858942ff2ade4539a7df784b81ec8769"
format_stamp: "Formatted at 2026-09-24 04:12:43 on dist-test-slave-00ld"
server_key: "57bf30f52d5c22b81172f64253f89a4c"
server_key_iv: "fe1205cacde407aa929f08cad9e00aa3"
server_key_version: "encryptionkey@0"
I20260924 04:12:43.877472 21526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790222898807720-392-0/minicluster-data/master-0/wal/instance:
uuid: "858942ff2ade4539a7df784b81ec8769"
format_stamp: "Formatted at 2026-09-24 04:12:43 on dist-test-slave-00ld"
server_key: "57bf30f52d5c22b81172f64253f89a4c"
server_key_iv: "fe1205cacde407aa929f08cad9e00aa3"
server_key_version: "encryptionkey@0"
I20260924 04:12:43.881083 21526 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:12:43.883419 21541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:12:43.884572 21526 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:12:43.884754 21526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790222898807720-392-0/minicluster-data/master-0/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790222898807720-392-0/minicluster-data/master-0/wal
uuid: "858942ff2ade4539a7df784b81ec8769"
format_stamp: "Formatted at 2026-09-24 04:12:43 on dist-test-slave-00ld"
server_key: "57bf30f52d5c22b81172f64253f89a4c"
server_key_iv: "fe1205cacde407aa929f08cad9e00aa3"
server_key_version: "encryptionkey@0"
I20260924 04:12:43.884881 21526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790222898807720-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790222898807720-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790222898807720-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:12:43.889381 21526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:12:43.892127 21526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:12:43.892318 21526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:12:43.898957 21526 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42007
I20260924 04:12:43.899015 21593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42007 every 8 connection(s)
I20260924 04:12:43.899860 21526 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:12:43.902518 21594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:12:43.907642 21594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 858942ff2ade4539a7df784b81ec8769: Bootstrap starting.
I20260924 04:12:43.909823 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 21526
I20260924 04:12:43.909965 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790222898807720-392-0/minicluster-data/master-0/wal/instance
I20260924 04:12:43.910161 21594 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 858942ff2ade4539a7df784b81ec8769: Neither blocks nor log segments found. Creating new log.
I20260924 04:12:43.910292 392 external_mini_cluster.cc:1428] Setting key 7d951adf077608923b58dc6879d2b066
I20260924 04:12:43.911023 21594 log.cc:824] T 00000000000000000000000000000000 P 858942ff2ade4539a7df784b81ec8769: Log is configured to *not* fsync() on all Append() calls
I20260924 04:12:43.913384 21594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 858942ff2ade4539a7df784b81ec8769: No bootstrap required, opened a new log
I20260924 04:12:43.915676 21598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:43.911394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.47:35589 (local address 127.0.0.1:42007)
0924 04:12:43.911842 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:43.911857 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:43.911922 (+ 65us) server_negotiation.cc:407] Connection header received
0924 04:12:43.912816 (+ 894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:43.912840 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:43.913186 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:43.913549 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:12:43.914293 (+ 744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:12:43.914298 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:12:43.914313 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:12:43.914325 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:12:43.914387 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:12:43.914414 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:12:43.914525 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:43.914605 (+ 80us) server_negotiation.cc:299] Negotiation successful
0924 04:12:43.914771 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":131,"threads_started":1}
I20260924 04:12:43.917965 21594 raft_consensus.cc:348] T 00000000000000000000000000000000 P 858942ff2ade4539a7df784b81ec8769 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "858942ff2ade4539a7df784b81ec8769" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42007 } }
I20260924 04:12:43.918232 21594 raft_consensus.cc:374] T 00000000000000000000000000000000 P 858942ff2ade4539a7df784b81ec8769 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:12:43.918313 21594 raft_consensus.cc:729] T 00000000000000000000000000000000 P 858942ff2ade4539a7df784b81ec8769 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 858942ff2ade4539a7df784b81ec8769, State: Initialized, Role: FOLLOWER
I20260924 04:12:43.918581 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:12:43.918809 21594 consensus_queue.cc:260] T 00000000000000000000000000000000 P 858942ff2ade4539a7df784b81ec8769 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "858942ff2ade4539a7df784b81ec8769" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42007 } }
I20260924 04:12:43.918972 21594 raft_consensus.cc:388] T 00000000000000000000000000000000 P 858942ff2ade4539a7df784b81ec8769 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:12:43.919057 21594 raft_consensus.cc:482] T 00000000000000000000000000000000 P 858942ff2ade4539a7df784b81ec8769 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:12:43.919160 21594 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 858942ff2ade4539a7df784b81ec8769 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:12:43.920120 21594 raft_consensus.cc:504] T 00000000000000000000000000000000 P 858942ff2ade4539a7df784b81ec8769 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "858942ff2ade4539a7df784b81ec8769" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42007 } }
I20260924 04:12:43.920454 21594 leader_election.cc:302] T 00000000000000000000000000000000 P 858942ff2ade4539a7df784b81ec8769 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 858942ff2ade4539a7df784b81ec8769; no voters:
I20260924 04:12:43.920830 21594 leader_election.cc:288] T 00000000000000000000000000000000 P 858942ff2ade4539a7df784b81ec8769 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:12:43.920892 21603 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 858942ff2ade4539a7df784b81ec8769 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:12:43.921115 21603 raft_consensus.cc:686] T 00000000000000000000000000000000 P 858942ff2ade4539a7df784b81ec8769 [term 1 LEADER]: Becoming Leader. State: Replica: 858942ff2ade4539a7df784b81ec8769, State: Running, Role: LEADER
I20260924 04:12:43.921439 21603 consensus_queue.cc:237] T 00000000000000000000000000000000 P 858942ff2ade4539a7df784b81ec8769 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "858942ff2ade4539a7df784b81ec8769" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42007 } }
I20260924 04:12:43.922107 21594 sys_catalog.cc:564] T 00000000000000000000000000000000 P 858942ff2ade4539a7df784b81ec8769 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:12:43.922578 21598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:43.920790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.47:48477 (local address 127.0.0.1:42007)
0924 04:12:43.920901 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:43.920906 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:43.921109 (+ 203us) server_negotiation.cc:407] Connection header received
0924 04:12:43.921275 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:43.921278 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:43.921309 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:43.921401 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:12:43.922127 (+ 726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:12:43.922130 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:12:43.922133 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:12:43.922144 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:12:43.922169 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:12:43.922207 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:12:43.922283 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:43.922423 (+ 140us) server_negotiation.cc:299] Negotiation successful
0924 04:12:43.922462 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 04:12:43.922643 21603 sys_catalog.cc:455] T 00000000000000000000000000000000 P 858942ff2ade4539a7df784b81ec8769 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 858942ff2ade4539a7df784b81ec8769. Latest consensus state: current_term: 1 leader_uuid: "858942ff2ade4539a7df784b81ec8769" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "858942ff2ade4539a7df784b81ec8769" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42007 } } }
I20260924 04:12:43.922798 21603 sys_catalog.cc:458] T 00000000000000000000000000000000 P 858942ff2ade4539a7df784b81ec8769 [sys.catalog]: This master's current role is: LEADER
I20260924 04:12:43.923028 21603 sys_catalog.cc:455] T 00000000000000000000000000000000 P 858942ff2ade4539a7df784b81ec8769 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "858942ff2ade4539a7df784b81ec8769" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "858942ff2ade4539a7df784b81ec8769" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42007 } } }
I20260924 04:12:43.923183 21603 sys_catalog.cc:458] T 00000000000000000000000000000000 P 858942ff2ade4539a7df784b81ec8769 [sys.catalog]: This master's current role is: LEADER
I20260924 04:12:43.923527 21611 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:12:43.926663 21611 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:12:43.932003 21611 catalog_manager.cc:1347] Generated new cluster ID: 7613568291eb406aaaa5cd2e4d5e4e4d
I20260924 04:12:43.932071 21611 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:12:43.962941 21611 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:12:43.963318 21611 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:12:43.969452 21611 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 858942ff2ade4539a7df784b81ec8769: Generated new TSK 0
I20260924 04:12:43.970184 21611 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:12:43.980005 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 21526
2026-09-24T04:12:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1429 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790222898807720-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:12:44 dist-test-slave-00ld krb5kdc[21625](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:12:44 dist-test-slave-00ld krb5kdc[21625](info): set up 2 sockets
Sep 24 04:12:44 dist-test-slave-00ld krb5kdc[21625](info): commencing operation
krb5kdc: starting...
W20260924 04:12:45.009100 392 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790222898807720-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790222898807720-392-0/krb5kdc/test-user.keytab.
Sep 24 04:12:45 dist-test-slave-00ld krb5kdc[21625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223165, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:12:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:12:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790222898807720-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790222898807720-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 04:12:45.130059 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790222898807720-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790222898807720-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790222898807720-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35749
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790222898807720-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35749
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:12:45.239804 21638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:12:45.240137 21638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:12:45.240226 21638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:12:45.244287 21638 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:12:45.244385 21638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:12:45.244424 21638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:12:45.244485 21638 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:12:45.244521 21638 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:12:45.250052 21638 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790222898807720-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790222898807720-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35749
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790222898807720-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35749
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21638
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790222898807720-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:12:45.251151 21638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:12:45.252189 21638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:12:45.260881 21646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:12:45.260881 21644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:12:45.260911 21638 server_base.cc:1034] running on GCE node
W20260924 04:12:45.260944 21643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:12:45.261751 21638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:12:45.262837 21638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:12:45.264043 21638 hybrid_clock.cc:648] HybridClock initialized: now 1790223165264014 us; error 43 us; skew 500 ppm
Sep 24 04:12:45 dist-test-slave-00ld krb5kdc[21625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223165, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:12:45.266892 21638 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 04:12:45.267823 21638 webserver.cc:460] Webserver started at http://127.0.0.1:46043/ using document root <none> and password file <none>
I20260924 04:12:45.268369 21638 fs_manager.cc:359] Metadata directory not provided
I20260924 04:12:45.268425 21638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:12:45.268586 21638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:12:45.270603 21638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790222898807720-392-0/minicluster-data/master-0/data/instance:
uuid: "dccea72bc3894a7386ebca29544ddb3b"
format_stamp: "Formatted at 2026-09-24 04:12:45 on dist-test-slave-00ld"
server_key: "15f2ac44cfd02a06311a191478739006"
server_key_iv: "9cf7e10959533c5aa70e57eb674057e1"
server_key_version: "encryptionkey@0"
I20260924 04:12:45.271104 21638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790222898807720-392-0/minicluster-data/master-0/wal/instance:
uuid: "dccea72bc3894a7386ebca29544ddb3b"
format_stamp: "Formatted at 2026-09-24 04:12:45 on dist-test-slave-00ld"
server_key: "15f2ac44cfd02a06311a191478739006"
server_key_iv: "9cf7e10959533c5aa70e57eb674057e1"
server_key_version: "encryptionkey@0"
I20260924 04:12:45.274650 21638 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 04:12:45.277071 21653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:12:45.278285 21638 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:12:45.278429 21638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790222898807720-392-0/minicluster-data/master-0/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790222898807720-392-0/minicluster-data/master-0/wal
uuid: "dccea72bc3894a7386ebca29544ddb3b"
format_stamp: "Formatted at 2026-09-24 04:12:45 on dist-test-slave-00ld"
server_key: "15f2ac44cfd02a06311a191478739006"
server_key_iv: "9cf7e10959533c5aa70e57eb674057e1"
server_key_version: "encryptionkey@0"
I20260924 04:12:45.278539 21638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790222898807720-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790222898807720-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790222898807720-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:12:45.291690 21638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:12:45.294807 21638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:12:45.295014 21638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:12:45.302423 21638 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35749
I20260924 04:12:45.302448 21705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35749 every 8 connection(s)
I20260924 04:12:45.303355 21638 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:12:45.306123 21706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:12:45.306159 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 21638
I20260924 04:12:45.306308 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790222898807720-392-0/minicluster-data/master-0/wal/instance
I20260924 04:12:45.306602 392 external_mini_cluster.cc:1428] Setting key 3fd8866ee5fa002c1b30333e5259ba2c
I20260924 04:12:45.312208 21706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dccea72bc3894a7386ebca29544ddb3b: Bootstrap starting.
I20260924 04:12:45.314363 21706 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dccea72bc3894a7386ebca29544ddb3b: Neither blocks nor log segments found. Creating new log.
I20260924 04:12:45.315049 21706 log.cc:824] T 00000000000000000000000000000000 P dccea72bc3894a7386ebca29544ddb3b: Log is configured to *not* fsync() on all Append() calls
Sep 24 04:12:45 dist-test-slave-00ld krb5kdc[21625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223165, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 04:12:45.317329 21706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dccea72bc3894a7386ebca29544ddb3b: No bootstrap required, opened a new log
I20260924 04:12:45.321050 21709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:45.307939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.47:54897 (local address 127.0.0.1:35749)
0924 04:12:45.308342 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:45.308357 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:45.308392 (+ 35us) server_negotiation.cc:407] Connection header received
0924 04:12:45.309224 (+ 832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:45.309243 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:45.309582 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:45.309908 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:12:45.310764 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:45.311745 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:45.312666 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:45.313192 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:45.313377 (+ 185us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:45.316031 (+ 2654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:12:45.316037 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:12:45.316054 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:12:45.316082 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:12:45.318001 (+ 1919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:12:45.318474 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:12:45.318480 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:12:45.318486 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:12:45.318629 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:12:45.318929 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:12:45.318932 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:12:45.318934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:12:45.319428 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:12:45.319670 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:45.319909 (+ 239us) server_negotiation.cc:299] Negotiation successful
0924 04:12:45.320127 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":118,"threads_started":1}
I20260924 04:12:45.321857 21706 raft_consensus.cc:348] T 00000000000000000000000000000000 P dccea72bc3894a7386ebca29544ddb3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dccea72bc3894a7386ebca29544ddb3b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35749 } }
I20260924 04:12:45.322114 21706 raft_consensus.cc:374] T 00000000000000000000000000000000 P dccea72bc3894a7386ebca29544ddb3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:12:45.322172 21706 raft_consensus.cc:729] T 00000000000000000000000000000000 P dccea72bc3894a7386ebca29544ddb3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dccea72bc3894a7386ebca29544ddb3b, State: Initialized, Role: FOLLOWER
I20260924 04:12:45.322578 21706 consensus_queue.cc:260] T 00000000000000000000000000000000 P dccea72bc3894a7386ebca29544ddb3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dccea72bc3894a7386ebca29544ddb3b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35749 } }
I20260924 04:12:45.322702 21706 raft_consensus.cc:388] T 00000000000000000000000000000000 P dccea72bc3894a7386ebca29544ddb3b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:12:45.322790 21706 raft_consensus.cc:482] T 00000000000000000000000000000000 P dccea72bc3894a7386ebca29544ddb3b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:12:45.322892 21706 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dccea72bc3894a7386ebca29544ddb3b [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:12:45.323930 21706 raft_consensus.cc:504] T 00000000000000000000000000000000 P dccea72bc3894a7386ebca29544ddb3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dccea72bc3894a7386ebca29544ddb3b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35749 } }
I20260924 04:12:45.324149 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:12:45.324339 21706 leader_election.cc:302] T 00000000000000000000000000000000 P dccea72bc3894a7386ebca29544ddb3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dccea72bc3894a7386ebca29544ddb3b; no voters:
I20260924 04:12:45.324663 21706 leader_election.cc:288] T 00000000000000000000000000000000 P dccea72bc3894a7386ebca29544ddb3b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:12:45.324766 21711 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dccea72bc3894a7386ebca29544ddb3b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:12:45.324960 21711 raft_consensus.cc:686] T 00000000000000000000000000000000 P dccea72bc3894a7386ebca29544ddb3b [term 1 LEADER]: Becoming Leader. State: Replica: dccea72bc3894a7386ebca29544ddb3b, State: Running, Role: LEADER
I20260924 04:12:45.325299 21711 consensus_queue.cc:237] T 00000000000000000000000000000000 P dccea72bc3894a7386ebca29544ddb3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dccea72bc3894a7386ebca29544ddb3b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35749 } }
I20260924 04:12:45.325937 21706 sys_catalog.cc:564] T 00000000000000000000000000000000 P dccea72bc3894a7386ebca29544ddb3b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:12:45.326977 21713 sys_catalog.cc:455] T 00000000000000000000000000000000 P dccea72bc3894a7386ebca29544ddb3b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dccea72bc3894a7386ebca29544ddb3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dccea72bc3894a7386ebca29544ddb3b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35749 } } }
I20260924 04:12:45.327121 21713 sys_catalog.cc:458] T 00000000000000000000000000000000 P dccea72bc3894a7386ebca29544ddb3b [sys.catalog]: This master's current role is: LEADER
I20260924 04:12:45.327302 21712 sys_catalog.cc:455] T 00000000000000000000000000000000 P dccea72bc3894a7386ebca29544ddb3b [sys.catalog]: SysCatalogTable state changed. Reason: New leader dccea72bc3894a7386ebca29544ddb3b. Latest consensus state: current_term: 1 leader_uuid: "dccea72bc3894a7386ebca29544ddb3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dccea72bc3894a7386ebca29544ddb3b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35749 } } }
I20260924 04:12:45.327458 21712 sys_catalog.cc:458] T 00000000000000000000000000000000 P dccea72bc3894a7386ebca29544ddb3b [sys.catalog]: This master's current role is: LEADER
I20260924 04:12:45.327752 21726 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:12:45.330984 21726 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:12:45.337258 21726 catalog_manager.cc:1347] Generated new cluster ID: 17da511f6d37418894a9a881ab479ea0
I20260924 04:12:45.337327 21726 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:12:45.337817 21709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:45.326248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.47:40283 (local address 127.0.0.1:35749)
0924 04:12:45.326364 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:45.326369 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:45.328991 (+ 2622us) server_negotiation.cc:407] Connection header received
0924 04:12:45.329186 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:45.329189 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:45.329236 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:45.329284 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:12:45.330215 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:45.330951 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:45.332099 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:45.332429 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:45.332504 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:45.334105 (+ 1601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:12:45.334108 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:12:45.334111 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:12:45.334136 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:12:45.335792 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:12:45.336284 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:12:45.336287 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:12:45.336289 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:12:45.336372 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:12:45.336937 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:12:45.336941 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:12:45.336943 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:12:45.337121 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:12:45.337276 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:45.337554 (+ 278us) server_negotiation.cc:299] Negotiation successful
0924 04:12:45.337650 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 04:12:45.349951 21726 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:12:45.351193 21726 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:12:45.356464 21726 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dccea72bc3894a7386ebca29544ddb3b: Generated new TSK 0
I20260924 04:12:45.357273 21726 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:12:45.363734 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 21638
W20260924 04:12:45.368397 21623 connection.cc:535] client connection to 127.0.0.1:35749 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35749): Connection reset by peer (error 104)
2026-09-24T04:12:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1382 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790222898807720-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:12:45 dist-test-slave-00ld krb5kdc[21737](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:12:45 dist-test-slave-00ld krb5kdc[21737](info): set up 2 sockets
Sep 24 04:12:45 dist-test-slave-00ld krb5kdc[21737](info): commencing operation
krb5kdc: starting...
W20260924 04:12:46.397150 392 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790222898807720-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790222898807720-392-0/krb5kdc/test-user.keytab.
Sep 24 04:12:46 dist-test-slave-00ld krb5kdc[21737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223166, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:12:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:12:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790222898807720-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790222898807720-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 04:12:46.524144 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790222898807720-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790222898807720-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790222898807720-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46365
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790222898807720-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46365
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:12:46.635030 21750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:12:46.635311 21750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:12:46.635353 21750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:12:46.639431 21750 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:12:46.639508 21750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:12:46.639528 21750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:12:46.639544 21750 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:12:46.639559 21750 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:12:46.644940 21750 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790222898807720-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790222898807720-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46365
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790222898807720-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21750
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790222898807720-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:12:46.646062 21750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:12:46.647143 21750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:12:46.656014 21758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:12:46.656032 21755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:12:46.656092 21756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:12:46.656185 21750 server_base.cc:1034] running on GCE node
I20260924 04:12:46.657061 21750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:12:46.658257 21750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:12:46.659456 21750 hybrid_clock.cc:648] HybridClock initialized: now 1790223166659432 us; error 39 us; skew 500 ppm
Sep 24 04:12:46 dist-test-slave-00ld krb5kdc[21737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223166, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:12:46.662293 21750 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 04:12:46.663201 21750 webserver.cc:460] Webserver started at http://127.0.0.1:36383/ using document root <none> and password file <none>
I20260924 04:12:46.663786 21750 fs_manager.cc:359] Metadata directory not provided
I20260924 04:12:46.663879 21750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:12:46.664115 21750 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:12:46.666265 21750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790222898807720-392-0/minicluster-data/master-0/data/instance:
uuid: "9135244bbcf74478b0575ae67efbb9d5"
format_stamp: "Formatted at 2026-09-24 04:12:46 on dist-test-slave-00ld"
server_key: "de2f32d93f2700ed0628f09379d75b0d"
server_key_iv: "22eecc3da797a573db44912ec40a9e52"
server_key_version: "encryptionkey@0"
I20260924 04:12:46.666858 21750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790222898807720-392-0/minicluster-data/master-0/wal/instance:
uuid: "9135244bbcf74478b0575ae67efbb9d5"
format_stamp: "Formatted at 2026-09-24 04:12:46 on dist-test-slave-00ld"
server_key: "de2f32d93f2700ed0628f09379d75b0d"
server_key_iv: "22eecc3da797a573db44912ec40a9e52"
server_key_version: "encryptionkey@0"
I20260924 04:12:46.670807 21750 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 04:12:46.673408 21765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:12:46.674705 21750 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:12:46.674866 21750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790222898807720-392-0/minicluster-data/master-0/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790222898807720-392-0/minicluster-data/master-0/wal
uuid: "9135244bbcf74478b0575ae67efbb9d5"
format_stamp: "Formatted at 2026-09-24 04:12:46 on dist-test-slave-00ld"
server_key: "de2f32d93f2700ed0628f09379d75b0d"
server_key_iv: "22eecc3da797a573db44912ec40a9e52"
server_key_version: "encryptionkey@0"
I20260924 04:12:46.674991 21750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790222898807720-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790222898807720-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790222898807720-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:12:46.695994 21750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:12:46.699225 21750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:12:46.699445 21750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:12:46.706728 21750 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46365
I20260924 04:12:46.706753 21817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46365 every 8 connection(s)
I20260924 04:12:46.707705 21750 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:12:46.710562 21818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:12:46.710753 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 21750
I20260924 04:12:46.710906 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790222898807720-392-0/minicluster-data/master-0/wal/instance
I20260924 04:12:46.711208 392 external_mini_cluster.cc:1428] Setting key f40518f3150d2ac72c02dab953fd7127
I20260924 04:12:46.717219 21818 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9135244bbcf74478b0575ae67efbb9d5: Bootstrap starting.
I20260924 04:12:46.719576 21818 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9135244bbcf74478b0575ae67efbb9d5: Neither blocks nor log segments found. Creating new log.
I20260924 04:12:46.720330 21818 log.cc:824] T 00000000000000000000000000000000 P 9135244bbcf74478b0575ae67efbb9d5: Log is configured to *not* fsync() on all Append() calls
I20260924 04:12:46.720764 21821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:46.712715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.47:33833 (local address 127.0.0.1:46365)
0924 04:12:46.713158 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:46.713174 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:46.713210 (+ 36us) server_negotiation.cc:407] Connection header received
0924 04:12:46.713931 (+ 721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:46.713949 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:46.714297 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:46.714663 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:12:46.715074 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:46.716092 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:46.717098 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:46.717637 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:46.717816 (+ 179us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:46.718809 (+ 993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:12:46.718814 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:12:46.718827 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:12:46.719244 (+ 417us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:12:46.719299 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:12:46.719322 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:12:46.719470 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:46.719654 (+ 184us) server_negotiation.cc:299] Negotiation successful
0924 04:12:46.719936 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":145,"threads_started":1}
I20260924 04:12:46.722260 21818 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9135244bbcf74478b0575ae67efbb9d5: No bootstrap required, opened a new log
I20260924 04:12:46.723625 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:12:46.726872 21818 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9135244bbcf74478b0575ae67efbb9d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9135244bbcf74478b0575ae67efbb9d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46365 } }
I20260924 04:12:46.727172 21818 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9135244bbcf74478b0575ae67efbb9d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:12:46.727221 21818 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9135244bbcf74478b0575ae67efbb9d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9135244bbcf74478b0575ae67efbb9d5, State: Initialized, Role: FOLLOWER
I20260924 04:12:46.727681 21818 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9135244bbcf74478b0575ae67efbb9d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9135244bbcf74478b0575ae67efbb9d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46365 } }
I20260924 04:12:46.727811 21818 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9135244bbcf74478b0575ae67efbb9d5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:12:46.727880 21818 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9135244bbcf74478b0575ae67efbb9d5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:12:46.728001 21818 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9135244bbcf74478b0575ae67efbb9d5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:12:46.729174 21818 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9135244bbcf74478b0575ae67efbb9d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9135244bbcf74478b0575ae67efbb9d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46365 } }
I20260924 04:12:46.729553 21818 leader_election.cc:302] T 00000000000000000000000000000000 P 9135244bbcf74478b0575ae67efbb9d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9135244bbcf74478b0575ae67efbb9d5; no voters:
I20260924 04:12:46.729905 21818 leader_election.cc:288] T 00000000000000000000000000000000 P 9135244bbcf74478b0575ae67efbb9d5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:12:46.730047 21828 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9135244bbcf74478b0575ae67efbb9d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:12:46.730393 21828 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9135244bbcf74478b0575ae67efbb9d5 [term 1 LEADER]: Becoming Leader. State: Replica: 9135244bbcf74478b0575ae67efbb9d5, State: Running, Role: LEADER
I20260924 04:12:46.730772 21828 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9135244bbcf74478b0575ae67efbb9d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9135244bbcf74478b0575ae67efbb9d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46365 } }
I20260924 04:12:46.731335 21818 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9135244bbcf74478b0575ae67efbb9d5 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:12:46.731400 21821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:46.725990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.47:33881 (local address 127.0.0.1:46365)
0924 04:12:46.726124 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:46.726129 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:46.726287 (+ 158us) server_negotiation.cc:407] Connection header received
0924 04:12:46.726384 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:46.726387 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:46.726433 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:46.726520 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:12:46.726749 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:46.727482 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:46.728508 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:46.728903 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:46.729032 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:46.730584 (+ 1552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:12:46.730587 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:12:46.730590 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:12:46.730631 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:12:46.730658 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:12:46.730671 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:12:46.730764 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:46.731161 (+ 397us) server_negotiation.cc:299] Negotiation successful
0924 04:12:46.731223 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 04:12:46.733912 21829 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9135244bbcf74478b0575ae67efbb9d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9135244bbcf74478b0575ae67efbb9d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9135244bbcf74478b0575ae67efbb9d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46365 } } }
I20260924 04:12:46.734043 21829 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9135244bbcf74478b0575ae67efbb9d5 [sys.catalog]: This master's current role is: LEADER
I20260924 04:12:46.734256 21829 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9135244bbcf74478b0575ae67efbb9d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9135244bbcf74478b0575ae67efbb9d5. Latest consensus state: current_term: 1 leader_uuid: "9135244bbcf74478b0575ae67efbb9d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9135244bbcf74478b0575ae67efbb9d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46365 } } }
I20260924 04:12:46.734324 21829 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9135244bbcf74478b0575ae67efbb9d5 [sys.catalog]: This master's current role is: LEADER
W20260924 04:12:46.736346 21842 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9135244bbcf74478b0575ae67efbb9d5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:12:46.736413 21842 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:12:46.736500 21843 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:12:46.737432 21843 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:12:46.743194 21843 catalog_manager.cc:1347] Generated new cluster ID: e26d17aaa95742488dff071a17227812
I20260924 04:12:46.743304 21843 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:12:46.763913 21843 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:12:46.765292 21843 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:12:46.775565 21843 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9135244bbcf74478b0575ae67efbb9d5: Generated new TSK 0
I20260924 04:12:46.776502 21843 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:12:46.788067 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 21750
W20260924 04:12:46.793637 21735 connection.cc:535] client connection to 127.0.0.1:46365 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46365): Connection reset by peer (error 104)
2026-09-24T04:12:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1425 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790222898807720-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:12:46 dist-test-slave-00ld krb5kdc[21849](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:12:46 dist-test-slave-00ld krb5kdc[21849](info): set up 2 sockets
Sep 24 04:12:46 dist-test-slave-00ld krb5kdc[21849](info): commencing operation
krb5kdc: starting...
W20260924 04:12:47.829066 392 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790222898807720-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790222898807720-392-0/krb5kdc/test-user.keytab.
Sep 24 04:12:47 dist-test-slave-00ld krb5kdc[21849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223167, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:12:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:12:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790222898807720-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790222898807720-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 04:12:47.952960 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790222898807720-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790222898807720-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790222898807720-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33857
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790222898807720-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33857
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:12:48.065300 21862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:12:48.065601 21862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:12:48.065644 21862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:12:48.069685 21862 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:12:48.069756 21862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:12:48.069775 21862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:12:48.069792 21862 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:12:48.069808 21862 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:12:48.075104 21862 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790222898807720-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790222898807720-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33857
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790222898807720-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21862
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790222898807720-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:12:48.076207 21862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:12:48.077257 21862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:12:48.085548 21870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:12:48.085613 21867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:12:48.085580 21862 server_base.cc:1034] running on GCE node
W20260924 04:12:48.085690 21868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:12:48.086354 21862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:12:48.087421 21862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:12:48.088582 21862 hybrid_clock.cc:648] HybridClock initialized: now 1790223168088496 us; error 101 us; skew 500 ppm
Sep 24 04:12:48 dist-test-slave-00ld krb5kdc[21849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223168, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:12:48.100162 21862 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 04:12:48.101241 21862 webserver.cc:460] Webserver started at http://127.0.0.1:32769/ using document root <none> and password file <none>
I20260924 04:12:48.101857 21862 fs_manager.cc:359] Metadata directory not provided
I20260924 04:12:48.101955 21862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:12:48.102195 21862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:12:48.104310 21862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790222898807720-392-0/minicluster-data/master-0/data/instance:
uuid: "b9560d5bcb454a0faed45218534085b3"
format_stamp: "Formatted at 2026-09-24 04:12:48 on dist-test-slave-00ld"
server_key: "c11902d1e42f5f464a50517d5e71414d"
server_key_iv: "afb19de1a52084a425f6aa7992cd6bf5"
server_key_version: "encryptionkey@0"
I20260924 04:12:48.104943 21862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790222898807720-392-0/minicluster-data/master-0/wal/instance:
uuid: "b9560d5bcb454a0faed45218534085b3"
format_stamp: "Formatted at 2026-09-24 04:12:48 on dist-test-slave-00ld"
server_key: "c11902d1e42f5f464a50517d5e71414d"
server_key_iv: "afb19de1a52084a425f6aa7992cd6bf5"
server_key_version: "encryptionkey@0"
I20260924 04:12:48.108640 21862 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260924 04:12:48.111112 21877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:12:48.112250 21862 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:12:48.112385 21862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790222898807720-392-0/minicluster-data/master-0/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790222898807720-392-0/minicluster-data/master-0/wal
uuid: "b9560d5bcb454a0faed45218534085b3"
format_stamp: "Formatted at 2026-09-24 04:12:48 on dist-test-slave-00ld"
server_key: "c11902d1e42f5f464a50517d5e71414d"
server_key_iv: "afb19de1a52084a425f6aa7992cd6bf5"
server_key_version: "encryptionkey@0"
I20260924 04:12:48.112491 21862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790222898807720-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790222898807720-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790222898807720-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:12:48.116899 21862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:12:48.119994 21862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:12:48.120188 21862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:12:48.126924 21862 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33857
I20260924 04:12:48.126950 21929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33857 every 8 connection(s)
I20260924 04:12:48.127785 21862 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:12:48.129357 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 21862
I20260924 04:12:48.129510 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790222898807720-392-0/minicluster-data/master-0/wal/instance
I20260924 04:12:48.129793 392 external_mini_cluster.cc:1428] Setting key eb3328fbce05756c607a7b57745b6b67
I20260924 04:12:48.130892 21930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:12:48.134639 21933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:48.131096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.47:48197 (local address 127.0.0.1:33857)
0924 04:12:48.131495 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:48.131510 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:48.131544 (+ 34us) server_negotiation.cc:407] Connection header received
0924 04:12:48.132146 (+ 602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:48.132162 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:48.132425 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:48.132766 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:12:48.133503 (+ 737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:12:48.133507 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:12:48.133518 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:12:48.133527 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:12:48.133575 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:12:48.133592 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:12:48.133710 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:48.133806 (+ 96us) server_negotiation.cc:299] Negotiation successful
0924 04:12:48.133965 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":123,"threads_started":1}
I20260924 04:12:48.137280 21930 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9560d5bcb454a0faed45218534085b3: Bootstrap starting.
I20260924 04:12:48.137473 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:12:48.140038 21930 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b9560d5bcb454a0faed45218534085b3: Neither blocks nor log segments found. Creating new log.
I20260924 04:12:48.140901 21930 log.cc:824] T 00000000000000000000000000000000 P b9560d5bcb454a0faed45218534085b3: Log is configured to *not* fsync() on all Append() calls
I20260924 04:12:48.141450 21933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:48.139770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.47:58559 (local address 127.0.0.1:33857)
0924 04:12:48.139907 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:48.139914 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:48.140081 (+ 167us) server_negotiation.cc:407] Connection header received
0924 04:12:48.140197 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:48.140200 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:48.140239 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:48.140331 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:12:48.141012 (+ 681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:12:48.141017 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:12:48.141022 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:12:48.141034 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:12:48.141061 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:12:48.141100 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:12:48.141216 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:48.141280 (+ 64us) server_negotiation.cc:299] Negotiation successful
0924 04:12:48.141321 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 04:12:48.142994 21930 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9560d5bcb454a0faed45218534085b3: No bootstrap required, opened a new log
I20260924 04:12:48.146536 21930 raft_consensus.cc:348] T 00000000000000000000000000000000 P b9560d5bcb454a0faed45218534085b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9560d5bcb454a0faed45218534085b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33857 } }
I20260924 04:12:48.146819 21930 raft_consensus.cc:374] T 00000000000000000000000000000000 P b9560d5bcb454a0faed45218534085b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:12:48.146942 21930 raft_consensus.cc:729] T 00000000000000000000000000000000 P b9560d5bcb454a0faed45218534085b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9560d5bcb454a0faed45218534085b3, State: Initialized, Role: FOLLOWER
I20260924 04:12:48.147416 21930 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9560d5bcb454a0faed45218534085b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9560d5bcb454a0faed45218534085b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33857 } }
I20260924 04:12:48.147590 21930 raft_consensus.cc:388] T 00000000000000000000000000000000 P b9560d5bcb454a0faed45218534085b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:12:48.147661 21930 raft_consensus.cc:482] T 00000000000000000000000000000000 P b9560d5bcb454a0faed45218534085b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:12:48.147758 21930 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b9560d5bcb454a0faed45218534085b3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:12:48.148665 21930 raft_consensus.cc:504] T 00000000000000000000000000000000 P b9560d5bcb454a0faed45218534085b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9560d5bcb454a0faed45218534085b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33857 } }
I20260924 04:12:48.149062 21930 leader_election.cc:302] T 00000000000000000000000000000000 P b9560d5bcb454a0faed45218534085b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b9560d5bcb454a0faed45218534085b3; no voters:
I20260924 04:12:48.149381 21930 leader_election.cc:288] T 00000000000000000000000000000000 P b9560d5bcb454a0faed45218534085b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:12:48.149529 21940 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b9560d5bcb454a0faed45218534085b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:12:48.149777 21940 raft_consensus.cc:686] T 00000000000000000000000000000000 P b9560d5bcb454a0faed45218534085b3 [term 1 LEADER]: Becoming Leader. State: Replica: b9560d5bcb454a0faed45218534085b3, State: Running, Role: LEADER
I20260924 04:12:48.150295 21940 consensus_queue.cc:237] T 00000000000000000000000000000000 P b9560d5bcb454a0faed45218534085b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9560d5bcb454a0faed45218534085b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33857 } }
I20260924 04:12:48.150677 21930 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9560d5bcb454a0faed45218534085b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:12:48.151921 21941 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9560d5bcb454a0faed45218534085b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b9560d5bcb454a0faed45218534085b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9560d5bcb454a0faed45218534085b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33857 } } }
I20260924 04:12:48.152051 21941 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9560d5bcb454a0faed45218534085b3 [sys.catalog]: This master's current role is: LEADER
I20260924 04:12:48.152424 21942 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9560d5bcb454a0faed45218534085b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9560d5bcb454a0faed45218534085b3. Latest consensus state: current_term: 1 leader_uuid: "b9560d5bcb454a0faed45218534085b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9560d5bcb454a0faed45218534085b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33857 } } }
I20260924 04:12:48.152510 21942 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9560d5bcb454a0faed45218534085b3 [sys.catalog]: This master's current role is: LEADER
I20260924 04:12:48.152846 21950 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:12:48.155777 21950 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:12:48.160949 21950 catalog_manager.cc:1347] Generated new cluster ID: 4e8d51297216426d8bc8b1aad352261a
I20260924 04:12:48.161023 21950 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:12:48.175019 21950 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:12:48.175390 21950 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:12:48.180055 21950 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b9560d5bcb454a0faed45218534085b3: Generated new TSK 0
I20260924 04:12:48.180742 21950 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:12:48.186535 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 21862
2026-09-24T04:12:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1398 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790222898807720-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:12:48 dist-test-slave-00ld krb5kdc[21961](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:12:48 dist-test-slave-00ld krb5kdc[21961](info): set up 2 sockets
Sep 24 04:12:48 dist-test-slave-00ld krb5kdc[21961](info): commencing operation
krb5kdc: starting...
W20260924 04:12:49.229004 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790222898807720-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790222898807720-392-0/krb5kdc/test-user.keytab.
Sep 24 04:12:49 dist-test-slave-00ld krb5kdc[21961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223169, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:12:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:12:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:12:49.355259 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790222898807720-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790222898807720-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790222898807720-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34125
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34125
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:12:49.465638 21974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:12:49.465972 21974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:12:49.466046 21974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:12:49.470115 21974 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:12:49.470217 21974 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:12:49.470256 21974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:12:49.470295 21974 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:12:49.470330 21974 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:12:49.475623 21974 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790222898807720-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790222898807720-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34125
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34125
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21974
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790222898807720-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:12:49.476703 21974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:12:49.477701 21974 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:12:49.485927 21979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:12:49.485930 21980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:12:49.485922 21982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:12:49.486135 21974 server_base.cc:1034] running on GCE node
I20260924 04:12:49.486703 21974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:12:49.487797 21974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:12:49.488953 21974 hybrid_clock.cc:648] HybridClock initialized: now 1790223169488935 us; error 27 us; skew 500 ppm
Sep 24 04:12:49 dist-test-slave-00ld krb5kdc[21961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223169, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:12:49.492002 21974 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:12:49.493019 21974 webserver.cc:460] Webserver started at http://127.0.98.62:33793/ using document root <none> and password file <none>
I20260924 04:12:49.493608 21974 fs_manager.cc:359] Metadata directory not provided
I20260924 04:12:49.493709 21974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:12:49.493942 21974 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:12:49.495944 21974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790222898807720-392-0/minicluster-data/master-0/data/instance:
uuid: "90b42a224c4c490789c5a885af8edf15"
format_stamp: "Formatted at 2026-09-24 04:12:49 on dist-test-slave-00ld"
server_key: "74bfb445f5de209ff4972d248ea1b762"
server_key_iv: "717d8ee8f5da2b90ea5537cc2beb98e5"
server_key_version: "encryptionkey@0"
I20260924 04:12:49.496531 21974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790222898807720-392-0/minicluster-data/master-0/wal/instance:
uuid: "90b42a224c4c490789c5a885af8edf15"
format_stamp: "Formatted at 2026-09-24 04:12:49 on dist-test-slave-00ld"
server_key: "74bfb445f5de209ff4972d248ea1b762"
server_key_iv: "717d8ee8f5da2b90ea5537cc2beb98e5"
server_key_version: "encryptionkey@0"
I20260924 04:12:49.500150 21974 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260924 04:12:49.502681 21989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:12:49.503898 21974 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:12:49.504042 21974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790222898807720-392-0/minicluster-data/master-0/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790222898807720-392-0/minicluster-data/master-0/wal
uuid: "90b42a224c4c490789c5a885af8edf15"
format_stamp: "Formatted at 2026-09-24 04:12:49 on dist-test-slave-00ld"
server_key: "74bfb445f5de209ff4972d248ea1b762"
server_key_iv: "717d8ee8f5da2b90ea5537cc2beb98e5"
server_key_version: "encryptionkey@0"
I20260924 04:12:49.504170 21974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790222898807720-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790222898807720-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790222898807720-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:12:49.516263 21974 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:12:49.519433 21974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:12:49.519644 21974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:12:49.526672 21974 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34125
I20260924 04:12:49.526695 22041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34125 every 8 connection(s)
I20260924 04:12:49.527606 21974 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:12:49.530491 22042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:12:49.531668 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 21974
I20260924 04:12:49.531831 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790222898807720-392-0/minicluster-data/master-0/wal/instance
I20260924 04:12:49.532127 392 external_mini_cluster.cc:1428] Setting key 5e959e6fdff40ab5debd070ea48b9d48
I20260924 04:12:49.537001 22042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90b42a224c4c490789c5a885af8edf15: Bootstrap starting.
I20260924 04:12:49.539539 22042 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 90b42a224c4c490789c5a885af8edf15: Neither blocks nor log segments found. Creating new log.
I20260924 04:12:49.540242 22042 log.cc:824] T 00000000000000000000000000000000 P 90b42a224c4c490789c5a885af8edf15: Log is configured to *not* fsync() on all Append() calls
Sep 24 04:12:49 dist-test-slave-00ld krb5kdc[21961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223169, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:12:49.542399 22042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90b42a224c4c490789c5a885af8edf15: No bootstrap required, opened a new log
I20260924 04:12:49.547194 22045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:49.533569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51749 (local address 127.0.98.62:34125)
0924 04:12:49.533978 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:49.533993 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:49.534029 (+ 36us) server_negotiation.cc:407] Connection header received
0924 04:12:49.534654 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:49.534671 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:49.535016 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:49.535326 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:12:49.536278 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:49.537401 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:49.538558 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:49.539062 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:49.539179 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:49.541974 (+ 2795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:12:49.541981 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:12:49.542001 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:12:49.542031 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:12:49.544037 (+ 2006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:12:49.544532 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:12:49.544539 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:12:49.544547 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:12:49.544729 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:12:49.545053 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:12:49.545058 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:12:49.545061 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:12:49.545570 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:12:49.545771 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:49.546003 (+ 232us) server_negotiation.cc:299] Negotiation successful
0924 04:12:49.546227 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":126,"threads_started":1}
I20260924 04:12:49.547190 22042 raft_consensus.cc:348] T 00000000000000000000000000000000 P 90b42a224c4c490789c5a885af8edf15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90b42a224c4c490789c5a885af8edf15" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34125 } }
I20260924 04:12:49.547520 22042 raft_consensus.cc:374] T 00000000000000000000000000000000 P 90b42a224c4c490789c5a885af8edf15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:12:49.547585 22042 raft_consensus.cc:729] T 00000000000000000000000000000000 P 90b42a224c4c490789c5a885af8edf15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90b42a224c4c490789c5a885af8edf15, State: Initialized, Role: FOLLOWER
I20260924 04:12:49.548055 22042 consensus_queue.cc:260] T 00000000000000000000000000000000 P 90b42a224c4c490789c5a885af8edf15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90b42a224c4c490789c5a885af8edf15" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34125 } }
I20260924 04:12:49.548213 22042 raft_consensus.cc:388] T 00000000000000000000000000000000 P 90b42a224c4c490789c5a885af8edf15 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:12:49.548341 22042 raft_consensus.cc:482] T 00000000000000000000000000000000 P 90b42a224c4c490789c5a885af8edf15 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:12:49.548480 22042 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 90b42a224c4c490789c5a885af8edf15 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:12:49.549603 22042 raft_consensus.cc:504] T 00000000000000000000000000000000 P 90b42a224c4c490789c5a885af8edf15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90b42a224c4c490789c5a885af8edf15" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34125 } }
I20260924 04:12:49.549995 22042 leader_election.cc:302] T 00000000000000000000000000000000 P 90b42a224c4c490789c5a885af8edf15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 90b42a224c4c490789c5a885af8edf15; no voters:
I20260924 04:12:49.550343 22042 leader_election.cc:288] T 00000000000000000000000000000000 P 90b42a224c4c490789c5a885af8edf15 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:12:49.550415 22047 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 90b42a224c4c490789c5a885af8edf15 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:12:49.550416 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:12:49.550622 22047 raft_consensus.cc:686] T 00000000000000000000000000000000 P 90b42a224c4c490789c5a885af8edf15 [term 1 LEADER]: Becoming Leader. State: Replica: 90b42a224c4c490789c5a885af8edf15, State: Running, Role: LEADER
I20260924 04:12:49.550988 22047 consensus_queue.cc:237] T 00000000000000000000000000000000 P 90b42a224c4c490789c5a885af8edf15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90b42a224c4c490789c5a885af8edf15" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34125 } }
I20260924 04:12:49.551666 22042 sys_catalog.cc:564] T 00000000000000000000000000000000 P 90b42a224c4c490789c5a885af8edf15 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:12:49.552604 22049 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90b42a224c4c490789c5a885af8edf15 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "90b42a224c4c490789c5a885af8edf15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90b42a224c4c490789c5a885af8edf15" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34125 } } }
I20260924 04:12:49.552788 22049 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90b42a224c4c490789c5a885af8edf15 [sys.catalog]: This master's current role is: LEADER
I20260924 04:12:49.553128 22048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90b42a224c4c490789c5a885af8edf15 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 90b42a224c4c490789c5a885af8edf15. Latest consensus state: current_term: 1 leader_uuid: "90b42a224c4c490789c5a885af8edf15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90b42a224c4c490789c5a885af8edf15" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34125 } } }
I20260924 04:12:49.553170 22060 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:12:49.553228 22048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90b42a224c4c490789c5a885af8edf15 [sys.catalog]: This master's current role is: LEADER
I20260924 04:12:49.556408 22060 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:12:49.563297 22045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:49.553819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44669 (local address 127.0.98.62:34125)
0924 04:12:49.553939 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:49.553958 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:49.554455 (+ 497us) server_negotiation.cc:407] Connection header received
0924 04:12:49.554606 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:49.554609 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:49.554653 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:49.554751 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:12:49.555594 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:49.556424 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:49.557556 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:49.557841 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:49.557943 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:49.559567 (+ 1624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:12:49.559571 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:12:49.559574 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:12:49.559600 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:12:49.561222 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:12:49.561798 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:12:49.561802 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:12:49.561804 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:12:49.561916 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:12:49.562319 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:12:49.562322 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:12:49.562324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:12:49.562551 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:12:49.562709 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:49.563024 (+ 315us) server_negotiation.cc:299] Negotiation successful
0924 04:12:49.563128 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 04:12:49.564109 22060 catalog_manager.cc:1347] Generated new cluster ID: 84e48fbb76d44a74b8fc5a4fc8d77cd2
I20260924 04:12:49.564234 22060 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:12:49.593024 22060 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:12:49.594347 22060 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:12:49.601850 22060 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 90b42a224c4c490789c5a885af8edf15: Generated new TSK 0
I20260924 04:12:49.602676 22060 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:12:49.609974 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 21974
W20260924 04:12:49.615520 21959 connection.cc:535] client connection to 127.0.98.62:34125 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:34125): Connection reset by peer (error 104)
2026-09-24T04:12:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1422 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790222898807720-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:12:49 dist-test-slave-00ld krb5kdc[22073](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:12:49 dist-test-slave-00ld krb5kdc[22073](info): set up 2 sockets
Sep 24 04:12:49 dist-test-slave-00ld krb5kdc[22073](info): commencing operation
krb5kdc: starting...
W20260924 04:12:50.640993 392 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790222898807720-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790222898807720-392-0/krb5kdc/test-user.keytab.
Sep 24 04:12:50 dist-test-slave-00ld krb5kdc[22073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223170, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:12:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:12:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:12:50.760679 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790222898807720-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790222898807720-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790222898807720-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38687
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38687
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:12:50.869778 22086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:12:50.870033 22086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:12:50.870074 22086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:12:50.874095 22086 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:12:50.874152 22086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:12:50.874171 22086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:12:50.874188 22086 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:12:50.874228 22086 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:12:50.879602 22086 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790222898807720-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790222898807720-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38687
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22086
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790222898807720-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:12:50.880611 22086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:12:50.881665 22086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:12:50.891877 22091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:12:50.891871 22094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:12:50.891871 22092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:12:50.892261 22086 server_base.cc:1034] running on GCE node
I20260924 04:12:50.892872 22086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:12:50.893963 22086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:12:50.895124 22086 hybrid_clock.cc:648] HybridClock initialized: now 1790223170895105 us; error 33 us; skew 500 ppm
Sep 24 04:12:50 dist-test-slave-00ld krb5kdc[22073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223170, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:12:50.898027 22086 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:12:50.898936 22086 webserver.cc:460] Webserver started at http://127.0.98.62:39447/ using document root <none> and password file <none>
I20260924 04:12:50.899488 22086 fs_manager.cc:359] Metadata directory not provided
I20260924 04:12:50.899549 22086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:12:50.899768 22086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:12:50.901878 22086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790222898807720-392-0/minicluster-data/master-0/data/instance:
uuid: "ea504679c81f4b579396e4608a12d083"
format_stamp: "Formatted at 2026-09-24 04:12:50 on dist-test-slave-00ld"
server_key: "bad83d1fa9971a6039555ee7038f1b9c"
server_key_iv: "03a48d23ff87cd1f84cda3d76b8c576c"
server_key_version: "encryptionkey@0"
I20260924 04:12:50.902439 22086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790222898807720-392-0/minicluster-data/master-0/wal/instance:
uuid: "ea504679c81f4b579396e4608a12d083"
format_stamp: "Formatted at 2026-09-24 04:12:50 on dist-test-slave-00ld"
server_key: "bad83d1fa9971a6039555ee7038f1b9c"
server_key_iv: "03a48d23ff87cd1f84cda3d76b8c576c"
server_key_version: "encryptionkey@0"
I20260924 04:12:50.906181 22086 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:12:50.908548 22101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:12:50.909838 22086 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:12:50.909964 22086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790222898807720-392-0/minicluster-data/master-0/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790222898807720-392-0/minicluster-data/master-0/wal
uuid: "ea504679c81f4b579396e4608a12d083"
format_stamp: "Formatted at 2026-09-24 04:12:50 on dist-test-slave-00ld"
server_key: "bad83d1fa9971a6039555ee7038f1b9c"
server_key_iv: "03a48d23ff87cd1f84cda3d76b8c576c"
server_key_version: "encryptionkey@0"
I20260924 04:12:50.910072 22086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790222898807720-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790222898807720-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790222898807720-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:12:50.930797 22086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:12:50.933924 22086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:12:50.934100 22086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:12:50.941044 22086 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38687
I20260924 04:12:50.941061 22153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38687 every 8 connection(s)
I20260924 04:12:50.941978 22086 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:12:50.944762 22154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:12:50.946718 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 22086
I20260924 04:12:50.946863 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790222898807720-392-0/minicluster-data/master-0/wal/instance
I20260924 04:12:50.947124 392 external_mini_cluster.cc:1428] Setting key 90f2173583bd304a137f74cd29a531b6
I20260924 04:12:50.951763 22154 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea504679c81f4b579396e4608a12d083: Bootstrap starting.
I20260924 04:12:50.954342 22154 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea504679c81f4b579396e4608a12d083: Neither blocks nor log segments found. Creating new log.
I20260924 04:12:50.955103 22154 log.cc:824] T 00000000000000000000000000000000 P ea504679c81f4b579396e4608a12d083: Log is configured to *not* fsync() on all Append() calls
I20260924 04:12:50.957273 22157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:50.948511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54757 (local address 127.0.98.62:38687)
0924 04:12:50.948991 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:50.949020 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:50.949062 (+ 42us) server_negotiation.cc:407] Connection header received
0924 04:12:50.949887 (+ 825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:50.949912 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:50.950357 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:50.950743 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:12:50.951153 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:50.952582 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:50.953531 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:50.954199 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:50.954367 (+ 168us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:50.955271 (+ 904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:12:50.955276 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:12:50.955292 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:12:50.955746 (+ 454us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:12:50.955834 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:12:50.955863 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:12:50.956023 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:50.956234 (+ 211us) server_negotiation.cc:299] Negotiation successful
0924 04:12:50.956354 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":178,"threads_started":1}
I20260924 04:12:50.957311 22154 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea504679c81f4b579396e4608a12d083: No bootstrap required, opened a new log
I20260924 04:12:50.960043 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:12:50.961134 22154 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea504679c81f4b579396e4608a12d083 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea504679c81f4b579396e4608a12d083" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38687 } }
I20260924 04:12:50.961443 22154 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea504679c81f4b579396e4608a12d083 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:12:50.961519 22154 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea504679c81f4b579396e4608a12d083 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea504679c81f4b579396e4608a12d083, State: Initialized, Role: FOLLOWER
I20260924 04:12:50.961958 22154 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea504679c81f4b579396e4608a12d083 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea504679c81f4b579396e4608a12d083" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38687 } }
I20260924 04:12:50.962113 22154 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea504679c81f4b579396e4608a12d083 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:12:50.962182 22154 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea504679c81f4b579396e4608a12d083 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:12:50.962296 22154 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea504679c81f4b579396e4608a12d083 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:12:50.963532 22154 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea504679c81f4b579396e4608a12d083 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea504679c81f4b579396e4608a12d083" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38687 } }
I20260924 04:12:50.963897 22154 leader_election.cc:302] T 00000000000000000000000000000000 P ea504679c81f4b579396e4608a12d083 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea504679c81f4b579396e4608a12d083; no voters:
I20260924 04:12:50.964231 22154 leader_election.cc:288] T 00000000000000000000000000000000 P ea504679c81f4b579396e4608a12d083 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:12:50.964299 22164 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea504679c81f4b579396e4608a12d083 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:12:50.964490 22164 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea504679c81f4b579396e4608a12d083 [term 1 LEADER]: Becoming Leader. State: Replica: ea504679c81f4b579396e4608a12d083, State: Running, Role: LEADER
I20260924 04:12:50.964862 22164 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea504679c81f4b579396e4608a12d083 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea504679c81f4b579396e4608a12d083" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38687 } }
I20260924 04:12:50.965576 22154 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea504679c81f4b579396e4608a12d083 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:12:50.966147 22164 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea504679c81f4b579396e4608a12d083 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea504679c81f4b579396e4608a12d083. Latest consensus state: current_term: 1 leader_uuid: "ea504679c81f4b579396e4608a12d083" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea504679c81f4b579396e4608a12d083" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38687 } } }
I20260924 04:12:50.966337 22164 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea504679c81f4b579396e4608a12d083 [sys.catalog]: This master's current role is: LEADER
I20260924 04:12:50.966614 22157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:50.962219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34351 (local address 127.0.98.62:38687)
0924 04:12:50.962382 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:50.962387 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:50.962498 (+ 111us) server_negotiation.cc:407] Connection header received
0924 04:12:50.962575 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:50.962578 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:50.962622 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:50.962711 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:12:50.962930 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:50.963675 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:50.964719 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:50.964979 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:50.965037 (+ 58us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:50.965984 (+ 947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:12:50.965988 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:12:50.965991 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:12:50.966003 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:12:50.966055 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:12:50.966070 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:12:50.966156 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:50.966392 (+ 236us) server_negotiation.cc:299] Negotiation successful
0924 04:12:50.966434 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 04:12:50.966580 22165 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea504679c81f4b579396e4608a12d083 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ea504679c81f4b579396e4608a12d083" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea504679c81f4b579396e4608a12d083" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38687 } } }
I20260924 04:12:50.966671 22165 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea504679c81f4b579396e4608a12d083 [sys.catalog]: This master's current role is: LEADER
I20260924 04:12:50.966964 22172 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:12:50.970053 22172 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:12:50.975157 22172 catalog_manager.cc:1347] Generated new cluster ID: 0520c6519b9147479afcfa980f448505
I20260924 04:12:50.975224 22172 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:12:51.019042 22172 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:12:51.020521 22172 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:12:51.028299 22172 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ea504679c81f4b579396e4608a12d083: Generated new TSK 0
I20260924 04:12:51.029247 22172 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:12:51.045076 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 22086
W20260924 04:12:51.053476 22071 connection.cc:535] client connection to 127.0.98.62:38687 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:38687): Connection reset by peer (error 104)
2026-09-24T04:12:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1437 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790222898807720-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:12:51 dist-test-slave-00ld krb5kdc[22185](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:12:51 dist-test-slave-00ld krb5kdc[22185](info): set up 2 sockets
Sep 24 04:12:51 dist-test-slave-00ld krb5kdc[22185](info): commencing operation
krb5kdc: starting...
W20260924 04:12:52.093004 392 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790222898807720-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790222898807720-392-0/krb5kdc/test-user.keytab.
Sep 24 04:12:52 dist-test-slave-00ld krb5kdc[22185](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223172, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:12:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:12:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:12:52.212183 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790222898807720-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790222898807720-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790222898807720-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44475
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44475
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:12:52.320502 22198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:12:52.320891 22198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:12:52.320972 22198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:12:52.324899 22198 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:12:52.324990 22198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:12:52.325029 22198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:12:52.325088 22198 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:12:52.325127 22198 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:12:52.330224 22198 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790222898807720-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790222898807720-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44475
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22198
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790222898807720-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:12:52.331243 22198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:12:52.332301 22198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:12:52.340663 22203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:12:52.340757 22204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:12:52.340911 22198 server_base.cc:1034] running on GCE node
W20260924 04:12:52.340662 22206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:12:52.341665 22198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:12:52.342967 22198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:12:52.344197 22198 hybrid_clock.cc:648] HybridClock initialized: now 1790223172344177 us; error 33 us; skew 500 ppm
Sep 24 04:12:52 dist-test-slave-00ld krb5kdc[22185](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223172, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:12:52.347378 22198 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:12:52.348417 22198 webserver.cc:460] Webserver started at http://127.0.98.62:36955/ using document root <none> and password file <none>
I20260924 04:12:52.349118 22198 fs_manager.cc:359] Metadata directory not provided
I20260924 04:12:52.349220 22198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:12:52.349469 22198 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:12:52.351678 22198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790222898807720-392-0/minicluster-data/master-0/data/instance:
uuid: "055cdbef471a4897975db30a54f39e71"
format_stamp: "Formatted at 2026-09-24 04:12:52 on dist-test-slave-00ld"
server_key: "20db6de4c6a503e6a6af2a316434d68e"
server_key_iv: "25638d7dd26b9d21177444ba3cfd46fc"
server_key_version: "encryptionkey@0"
I20260924 04:12:52.352330 22198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790222898807720-392-0/minicluster-data/master-0/wal/instance:
uuid: "055cdbef471a4897975db30a54f39e71"
format_stamp: "Formatted at 2026-09-24 04:12:52 on dist-test-slave-00ld"
server_key: "20db6de4c6a503e6a6af2a316434d68e"
server_key_iv: "25638d7dd26b9d21177444ba3cfd46fc"
server_key_version: "encryptionkey@0"
I20260924 04:12:52.356516 22198 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260924 04:12:52.359665 22213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:12:52.361080 22198 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:12:52.361203 22198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790222898807720-392-0/minicluster-data/master-0/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790222898807720-392-0/minicluster-data/master-0/wal
uuid: "055cdbef471a4897975db30a54f39e71"
format_stamp: "Formatted at 2026-09-24 04:12:52 on dist-test-slave-00ld"
server_key: "20db6de4c6a503e6a6af2a316434d68e"
server_key_iv: "25638d7dd26b9d21177444ba3cfd46fc"
server_key_version: "encryptionkey@0"
I20260924 04:12:52.361296 22198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790222898807720-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790222898807720-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790222898807720-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:12:52.365912 22198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:12:52.368992 22198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:12:52.369143 22198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:12:52.376279 22198 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44475
I20260924 04:12:52.376297 22265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44475 every 8 connection(s)
I20260924 04:12:52.377275 22198 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:12:52.377513 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 22198
I20260924 04:12:52.377662 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790222898807720-392-0/minicluster-data/master-0/wal/instance
I20260924 04:12:52.377990 392 external_mini_cluster.cc:1428] Setting key 0af147ceec8f29cc8c85001b4e1efca4
I20260924 04:12:52.380383 22266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:12:52.383036 22269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:52.379364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58603 (local address 127.0.98.62:44475)
0924 04:12:52.379767 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:52.379781 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:52.379814 (+ 33us) server_negotiation.cc:407] Connection header received
0924 04:12:52.380465 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:52.380490 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:52.380862 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:52.381191 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:12:52.381930 (+ 739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:12:52.381934 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:12:52.381946 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:12:52.381955 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:12:52.382003 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:12:52.382022 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:12:52.382164 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:52.382233 (+ 69us) server_negotiation.cc:299] Negotiation successful
0924 04:12:52.382383 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":144,"threads_started":1}
I20260924 04:12:52.385875 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:12:52.386518 22266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 055cdbef471a4897975db30a54f39e71: Bootstrap starting.
I20260924 04:12:52.388902 22266 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 055cdbef471a4897975db30a54f39e71: Neither blocks nor log segments found. Creating new log.
I20260924 04:12:52.389626 22269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:52.387881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33803 (local address 127.0.98.62:44475)
0924 04:12:52.387987 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:52.387990 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:52.388200 (+ 210us) server_negotiation.cc:407] Connection header received
0924 04:12:52.388341 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:52.388344 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:52.388383 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:52.388463 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:12:52.389244 (+ 781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:12:52.389248 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:12:52.389250 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:12:52.389261 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:12:52.389285 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:12:52.389296 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:12:52.389387 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:52.389467 (+ 80us) server_negotiation.cc:299] Negotiation successful
0924 04:12:52.389503 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 04:12:52.389644 22266 log.cc:824] T 00000000000000000000000000000000 P 055cdbef471a4897975db30a54f39e71: Log is configured to *not* fsync() on all Append() calls
I20260924 04:12:52.391639 22266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 055cdbef471a4897975db30a54f39e71: No bootstrap required, opened a new log
I20260924 04:12:52.395036 22266 raft_consensus.cc:348] T 00000000000000000000000000000000 P 055cdbef471a4897975db30a54f39e71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "055cdbef471a4897975db30a54f39e71" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44475 } }
I20260924 04:12:52.395258 22266 raft_consensus.cc:374] T 00000000000000000000000000000000 P 055cdbef471a4897975db30a54f39e71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:12:52.395345 22266 raft_consensus.cc:729] T 00000000000000000000000000000000 P 055cdbef471a4897975db30a54f39e71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 055cdbef471a4897975db30a54f39e71, State: Initialized, Role: FOLLOWER
I20260924 04:12:52.395803 22266 consensus_queue.cc:260] T 00000000000000000000000000000000 P 055cdbef471a4897975db30a54f39e71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "055cdbef471a4897975db30a54f39e71" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44475 } }
I20260924 04:12:52.395924 22266 raft_consensus.cc:388] T 00000000000000000000000000000000 P 055cdbef471a4897975db30a54f39e71 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:12:52.396036 22266 raft_consensus.cc:482] T 00000000000000000000000000000000 P 055cdbef471a4897975db30a54f39e71 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:12:52.396204 22266 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 055cdbef471a4897975db30a54f39e71 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:12:52.397346 22266 raft_consensus.cc:504] T 00000000000000000000000000000000 P 055cdbef471a4897975db30a54f39e71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "055cdbef471a4897975db30a54f39e71" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44475 } }
I20260924 04:12:52.397660 22266 leader_election.cc:302] T 00000000000000000000000000000000 P 055cdbef471a4897975db30a54f39e71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 055cdbef471a4897975db30a54f39e71; no voters:
I20260924 04:12:52.397986 22266 leader_election.cc:288] T 00000000000000000000000000000000 P 055cdbef471a4897975db30a54f39e71 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:12:52.398041 22276 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 055cdbef471a4897975db30a54f39e71 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:12:52.398232 22276 raft_consensus.cc:686] T 00000000000000000000000000000000 P 055cdbef471a4897975db30a54f39e71 [term 1 LEADER]: Becoming Leader. State: Replica: 055cdbef471a4897975db30a54f39e71, State: Running, Role: LEADER
I20260924 04:12:52.398554 22276 consensus_queue.cc:237] T 00000000000000000000000000000000 P 055cdbef471a4897975db30a54f39e71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "055cdbef471a4897975db30a54f39e71" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44475 } }
I20260924 04:12:52.399238 22266 sys_catalog.cc:564] T 00000000000000000000000000000000 P 055cdbef471a4897975db30a54f39e71 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:12:52.400287 22278 sys_catalog.cc:455] T 00000000000000000000000000000000 P 055cdbef471a4897975db30a54f39e71 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "055cdbef471a4897975db30a54f39e71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "055cdbef471a4897975db30a54f39e71" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44475 } } }
I20260924 04:12:52.400420 22278 sys_catalog.cc:458] T 00000000000000000000000000000000 P 055cdbef471a4897975db30a54f39e71 [sys.catalog]: This master's current role is: LEADER
I20260924 04:12:52.400849 22285 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:12:52.401162 22277 sys_catalog.cc:455] T 00000000000000000000000000000000 P 055cdbef471a4897975db30a54f39e71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 055cdbef471a4897975db30a54f39e71. Latest consensus state: current_term: 1 leader_uuid: "055cdbef471a4897975db30a54f39e71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "055cdbef471a4897975db30a54f39e71" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44475 } } }
I20260924 04:12:52.401316 22277 sys_catalog.cc:458] T 00000000000000000000000000000000 P 055cdbef471a4897975db30a54f39e71 [sys.catalog]: This master's current role is: LEADER
I20260924 04:12:52.404017 22285 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:12:52.409550 22285 catalog_manager.cc:1347] Generated new cluster ID: 7b09106f44104ccb82ec9710dc5c2633
I20260924 04:12:52.409643 22285 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:12:52.421584 22285 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:12:52.421880 22285 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:12:52.427098 22285 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 055cdbef471a4897975db30a54f39e71: Generated new TSK 0
I20260924 04:12:52.427893 22285 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:12:52.442570 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 22198
2026-09-24T04:12:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1394 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790222898807720-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:12:52 dist-test-slave-00ld krb5kdc[22297](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:12:52 dist-test-slave-00ld krb5kdc[22297](info): set up 2 sockets
Sep 24 04:12:52 dist-test-slave-00ld krb5kdc[22297](info): commencing operation
krb5kdc: starting...
W20260924 04:12:53.480959 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790222898807720-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790222898807720-392-0/krb5kdc/test-user.keytab.
Sep 24 04:12:53 dist-test-slave-00ld krb5kdc[22297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223173, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:12:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:12:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:12:53.604285 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790222898807720-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790222898807720-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790222898807720-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36287
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36287
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:12:53.714295 22310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:12:53.714567 22310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:12:53.714610 22310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:12:53.718677 22310 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:12:53.718745 22310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:12:53.718765 22310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:12:53.718781 22310 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:12:53.718796 22310 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:12:53.724117 22310 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790222898807720-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790222898807720-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36287
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36287
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22310
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790222898807720-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:12:53.725208 22310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:12:53.726290 22310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:12:53.734833 22318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:12:53.734848 22315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:12:53.734830 22316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:12:53.735205 22310 server_base.cc:1034] running on GCE node
I20260924 04:12:53.735761 22310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:12:53.736935 22310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:12:53.738118 22310 hybrid_clock.cc:648] HybridClock initialized: now 1790223173738099 us; error 35 us; skew 500 ppm
Sep 24 04:12:53 dist-test-slave-00ld krb5kdc[22297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223173, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:12:53.747193 22310 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:12:53.748245 22310 webserver.cc:460] Webserver started at http://127.0.98.62:43641/ using document root <none> and password file <none>
I20260924 04:12:53.748919 22310 fs_manager.cc:359] Metadata directory not provided
I20260924 04:12:53.749012 22310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:12:53.749270 22310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:12:53.751408 22310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790222898807720-392-0/minicluster-data/master-0/data/instance:
uuid: "f6cd9dd4cf2f41bfbf0627b77d0ae551"
format_stamp: "Formatted at 2026-09-24 04:12:53 on dist-test-slave-00ld"
server_key: "462fdcd943c8a90619d550793e6cfe02"
server_key_iv: "ce2b48e0eca47476a5b1ceb50adaac7a"
server_key_version: "encryptionkey@0"
I20260924 04:12:53.751971 22310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790222898807720-392-0/minicluster-data/master-0/wal/instance:
uuid: "f6cd9dd4cf2f41bfbf0627b77d0ae551"
format_stamp: "Formatted at 2026-09-24 04:12:53 on dist-test-slave-00ld"
server_key: "462fdcd943c8a90619d550793e6cfe02"
server_key_iv: "ce2b48e0eca47476a5b1ceb50adaac7a"
server_key_version: "encryptionkey@0"
I20260924 04:12:53.755716 22310 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 04:12:53.758589 22325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:12:53.760175 22310 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 04:12:53.760299 22310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790222898807720-392-0/minicluster-data/master-0/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790222898807720-392-0/minicluster-data/master-0/wal
uuid: "f6cd9dd4cf2f41bfbf0627b77d0ae551"
format_stamp: "Formatted at 2026-09-24 04:12:53 on dist-test-slave-00ld"
server_key: "462fdcd943c8a90619d550793e6cfe02"
server_key_iv: "ce2b48e0eca47476a5b1ceb50adaac7a"
server_key_version: "encryptionkey@0"
I20260924 04:12:53.760422 22310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790222898807720-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790222898807720-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790222898807720-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:12:53.770910 22310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:12:53.773541 22310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:12:53.773695 22310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:12:53.780552 22310 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36287
I20260924 04:12:53.780582 22377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36287 every 8 connection(s)
I20260924 04:12:53.781494 22310 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:12:53.784158 22378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:12:53.789420 22378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6cd9dd4cf2f41bfbf0627b77d0ae551: Bootstrap starting.
I20260924 04:12:53.790657 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 22310
I20260924 04:12:53.790822 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790222898807720-392-0/minicluster-data/master-0/wal/instance
I20260924 04:12:53.791118 392 external_mini_cluster.cc:1428] Setting key 6c05f6f369e2832c33ff7a531446d428
I20260924 04:12:53.792124 22378 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f6cd9dd4cf2f41bfbf0627b77d0ae551: Neither blocks nor log segments found. Creating new log.
I20260924 04:12:53.792987 22378 log.cc:824] T 00000000000000000000000000000000 P f6cd9dd4cf2f41bfbf0627b77d0ae551: Log is configured to *not* fsync() on all Append() calls
I20260924 04:12:53.795159 22378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6cd9dd4cf2f41bfbf0627b77d0ae551: No bootstrap required, opened a new log
I20260924 04:12:53.799355 22378 raft_consensus.cc:348] T 00000000000000000000000000000000 P f6cd9dd4cf2f41bfbf0627b77d0ae551 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6cd9dd4cf2f41bfbf0627b77d0ae551" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36287 } }
I20260924 04:12:53.799571 22378 raft_consensus.cc:374] T 00000000000000000000000000000000 P f6cd9dd4cf2f41bfbf0627b77d0ae551 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:12:53.799640 22378 raft_consensus.cc:729] T 00000000000000000000000000000000 P f6cd9dd4cf2f41bfbf0627b77d0ae551 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6cd9dd4cf2f41bfbf0627b77d0ae551, State: Initialized, Role: FOLLOWER
I20260924 04:12:53.800144 22378 consensus_queue.cc:260] T 00000000000000000000000000000000 P f6cd9dd4cf2f41bfbf0627b77d0ae551 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6cd9dd4cf2f41bfbf0627b77d0ae551" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36287 } }
I20260924 04:12:53.800268 22378 raft_consensus.cc:388] T 00000000000000000000000000000000 P f6cd9dd4cf2f41bfbf0627b77d0ae551 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:12:53.800354 22378 raft_consensus.cc:482] T 00000000000000000000000000000000 P f6cd9dd4cf2f41bfbf0627b77d0ae551 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:12:53.800477 22378 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f6cd9dd4cf2f41bfbf0627b77d0ae551 [term 0 FOLLOWER]: Advancing to term 1
Sep 24 04:12:53 dist-test-slave-00ld krb5kdc[22297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223173, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:12:53.801514 22378 raft_consensus.cc:504] T 00000000000000000000000000000000 P f6cd9dd4cf2f41bfbf0627b77d0ae551 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6cd9dd4cf2f41bfbf0627b77d0ae551" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36287 } }
I20260924 04:12:53.801894 22378 leader_election.cc:302] T 00000000000000000000000000000000 P f6cd9dd4cf2f41bfbf0627b77d0ae551 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f6cd9dd4cf2f41bfbf0627b77d0ae551; no voters:
I20260924 04:12:53.802323 22378 leader_election.cc:288] T 00000000000000000000000000000000 P f6cd9dd4cf2f41bfbf0627b77d0ae551 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:12:53.802388 22383 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f6cd9dd4cf2f41bfbf0627b77d0ae551 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:12:53.802618 22383 raft_consensus.cc:686] T 00000000000000000000000000000000 P f6cd9dd4cf2f41bfbf0627b77d0ae551 [term 1 LEADER]: Becoming Leader. State: Replica: f6cd9dd4cf2f41bfbf0627b77d0ae551, State: Running, Role: LEADER
I20260924 04:12:53.802909 22383 consensus_queue.cc:237] T 00000000000000000000000000000000 P f6cd9dd4cf2f41bfbf0627b77d0ae551 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6cd9dd4cf2f41bfbf0627b77d0ae551" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36287 } }
I20260924 04:12:53.803632 22378 sys_catalog.cc:564] T 00000000000000000000000000000000 P f6cd9dd4cf2f41bfbf0627b77d0ae551 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:12:53.804064 22385 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6cd9dd4cf2f41bfbf0627b77d0ae551 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f6cd9dd4cf2f41bfbf0627b77d0ae551" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6cd9dd4cf2f41bfbf0627b77d0ae551" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36287 } } }
I20260924 04:12:53.804209 22385 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6cd9dd4cf2f41bfbf0627b77d0ae551 [sys.catalog]: This master's current role is: LEADER
I20260924 04:12:53.804543 22384 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6cd9dd4cf2f41bfbf0627b77d0ae551 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f6cd9dd4cf2f41bfbf0627b77d0ae551. Latest consensus state: current_term: 1 leader_uuid: "f6cd9dd4cf2f41bfbf0627b77d0ae551" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6cd9dd4cf2f41bfbf0627b77d0ae551" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36287 } } }
I20260924 04:12:53.804672 22384 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6cd9dd4cf2f41bfbf0627b77d0ae551 [sys.catalog]: This master's current role is: LEADER
I20260924 04:12:53.807964 22381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:53.792306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54905 (local address 127.0.98.62:36287)
0924 04:12:53.792797 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:53.792816 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:53.792859 (+ 43us) server_negotiation.cc:407] Connection header received
0924 04:12:53.793656 (+ 797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:53.793674 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:53.794097 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:53.794451 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:12:53.795194 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:53.796744 (+ 1550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:53.797772 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:53.798600 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:53.798765 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:53.801791 (+ 3026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:12:53.801797 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:12:53.801813 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:12:53.801844 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:12:53.804343 (+ 2499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:12:53.805016 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:12:53.805023 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:12:53.805030 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:12:53.805204 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:12:53.805551 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:12:53.805554 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:12:53.805556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:12:53.806312 (+ 756us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:12:53.806541 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:53.806840 (+ 299us) server_negotiation.cc:299] Negotiation successful
0924 04:12:53.807202 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":143,"threads_started":1}
W20260924 04:12:53.808653 22398 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f6cd9dd4cf2f41bfbf0627b77d0ae551: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:12:53.808759 22398 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:12:53.808822 22387 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:12:53.809652 22387 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:12:53.811035 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:12:53.816407 22387 catalog_manager.cc:1347] Generated new cluster ID: eca9402dfc104579a083d6db322053e2
I20260924 04:12:53.816478 22387 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:12:53.827541 22381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:53.816281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53741 (local address 127.0.98.62:36287)
0924 04:12:53.816390 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:53.816394 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:53.816409 (+ 15us) server_negotiation.cc:407] Connection header received
0924 04:12:53.816451 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:53.816456 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:53.816510 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:53.817551 (+ 1041us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:12:53.817648 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:53.818499 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:53.819755 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:53.820152 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:53.820612 (+ 460us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:53.823082 (+ 2470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:12:53.823086 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:12:53.823089 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:12:53.823151 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:12:53.825768 (+ 2617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:12:53.826261 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:12:53.826264 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:12:53.826266 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:12:53.826341 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:12:53.826663 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:12:53.826667 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:12:53.826669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:12:53.826882 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:12:53.827006 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:53.827259 (+ 253us) server_negotiation.cc:299] Negotiation successful
0924 04:12:53.827377 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 04:12:53.832726 22387 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:12:53.833959 22387 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:12:53.848044 22387 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f6cd9dd4cf2f41bfbf0627b77d0ae551: Generated new TSK 0
I20260924 04:12:53.849025 22387 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:12:53.854058 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 22310
2026-09-24T04:12:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1411 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790222898807720-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:12:53 dist-test-slave-00ld krb5kdc[22409](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:12:53 dist-test-slave-00ld krb5kdc[22409](info): set up 2 sockets
Sep 24 04:12:53 dist-test-slave-00ld krb5kdc[22409](info): commencing operation
krb5kdc: starting...
W20260924 04:12:54.889024 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790222898807720-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790222898807720-392-0/krb5kdc/test-user.keytab.
Sep 24 04:12:54 dist-test-slave-00ld krb5kdc[22409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223174, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:12:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:12:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:12:55.011909 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790222898807720-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790222898807720-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790222898807720-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39241
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39241
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:12:55.121424 22422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:12:55.121688 22422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:12:55.121730 22422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:12:55.125679 22422 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:12:55.125735 22422 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:12:55.125754 22422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:12:55.125771 22422 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:12:55.125787 22422 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:12:55.130887 22422 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790222898807720-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790222898807720-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39241
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22422
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790222898807720-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:12:55.131820 22422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:12:55.132761 22422 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:12:55.141247 22430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:12:55.141268 22428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:12:55.141247 22427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:12:55.141439 22422 server_base.cc:1034] running on GCE node
I20260924 04:12:55.142094 22422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:12:55.143333 22422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:12:55.144510 22422 hybrid_clock.cc:648] HybridClock initialized: now 1790223175144488 us; error 44 us; skew 500 ppm
Sep 24 04:12:55 dist-test-slave-00ld krb5kdc[22409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223175, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:12:55.147454 22422 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:12:55.148306 22422 webserver.cc:460] Webserver started at http://127.0.98.62:44473/ using document root <none> and password file <none>
I20260924 04:12:55.148939 22422 fs_manager.cc:359] Metadata directory not provided
I20260924 04:12:55.149003 22422 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:12:55.149174 22422 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:12:55.151183 22422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790222898807720-392-0/minicluster-data/master-0/data/instance:
uuid: "4e82ff91d18b4f17b819e366f95ef752"
format_stamp: "Formatted at 2026-09-24 04:12:55 on dist-test-slave-00ld"
server_key: "c7bf117a0df4eb3774cdb5c72b79fe64"
server_key_iv: "d6f412556b108896ea90f9a672298e8d"
server_key_version: "encryptionkey@0"
I20260924 04:12:55.151695 22422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790222898807720-392-0/minicluster-data/master-0/wal/instance:
uuid: "4e82ff91d18b4f17b819e366f95ef752"
format_stamp: "Formatted at 2026-09-24 04:12:55 on dist-test-slave-00ld"
server_key: "c7bf117a0df4eb3774cdb5c72b79fe64"
server_key_iv: "d6f412556b108896ea90f9a672298e8d"
server_key_version: "encryptionkey@0"
I20260924 04:12:55.155339 22422 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 04:12:55.157792 22437 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:12:55.159173 22422 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 04:12:55.159317 22422 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790222898807720-392-0/minicluster-data/master-0/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790222898807720-392-0/minicluster-data/master-0/wal
uuid: "4e82ff91d18b4f17b819e366f95ef752"
format_stamp: "Formatted at 2026-09-24 04:12:55 on dist-test-slave-00ld"
server_key: "c7bf117a0df4eb3774cdb5c72b79fe64"
server_key_iv: "d6f412556b108896ea90f9a672298e8d"
server_key_version: "encryptionkey@0"
I20260924 04:12:55.159416 22422 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790222898807720-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790222898807720-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790222898807720-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:12:55.176623 22422 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:12:55.179821 22422 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:12:55.179996 22422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:12:55.187506 22489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39241 every 8 connection(s)
I20260924 04:12:55.187510 22422 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39241
I20260924 04:12:55.188661 22422 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:12:55.191499 22490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:12:55.196954 22490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e82ff91d18b4f17b819e366f95ef752: Bootstrap starting.
I20260924 04:12:55.198361 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 22422
I20260924 04:12:55.198546 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790222898807720-392-0/minicluster-data/master-0/wal/instance
I20260924 04:12:55.198853 392 external_mini_cluster.cc:1428] Setting key ed953b5027dec11d5ee79fed0153d44e
I20260924 04:12:55.199532 22490 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e82ff91d18b4f17b819e366f95ef752: Neither blocks nor log segments found. Creating new log.
I20260924 04:12:55.200462 22490 log.cc:824] T 00000000000000000000000000000000 P 4e82ff91d18b4f17b819e366f95ef752: Log is configured to *not* fsync() on all Append() calls
I20260924 04:12:55.202554 22490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e82ff91d18b4f17b819e366f95ef752: No bootstrap required, opened a new log
I20260924 04:12:55.206771 22490 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e82ff91d18b4f17b819e366f95ef752 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e82ff91d18b4f17b819e366f95ef752" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39241 } }
I20260924 04:12:55.206983 22490 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4e82ff91d18b4f17b819e366f95ef752 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:12:55.207016 22490 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e82ff91d18b4f17b819e366f95ef752 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e82ff91d18b4f17b819e366f95ef752, State: Initialized, Role: FOLLOWER
I20260924 04:12:55.207531 22490 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e82ff91d18b4f17b819e366f95ef752 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e82ff91d18b4f17b819e366f95ef752" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39241 } }
I20260924 04:12:55.207773 22490 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4e82ff91d18b4f17b819e366f95ef752 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:12:55.207849 22490 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e82ff91d18b4f17b819e366f95ef752 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:12:55.207947 22490 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e82ff91d18b4f17b819e366f95ef752 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:12:55.208643 22493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:55.200274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58279 (local address 127.0.98.62:39241)
0924 04:12:55.200760 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:55.200777 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:55.200813 (+ 36us) server_negotiation.cc:407] Connection header received
0924 04:12:55.201654 (+ 841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:55.201675 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:55.202069 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:55.202379 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:12:55.202813 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:55.203849 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:55.205186 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:55.205664 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:55.205796 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:55.206645 (+ 849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:12:55.206650 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:12:55.206664 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:12:55.207153 (+ 489us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:12:55.207208 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:12:55.207232 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:12:55.207361 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:55.207507 (+ 146us) server_negotiation.cc:299] Negotiation successful
0924 04:12:55.207787 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":150,"threads_started":1}
I20260924 04:12:55.208911 22490 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e82ff91d18b4f17b819e366f95ef752 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e82ff91d18b4f17b819e366f95ef752" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39241 } }
I20260924 04:12:55.209255 22490 leader_election.cc:302] T 00000000000000000000000000000000 P 4e82ff91d18b4f17b819e366f95ef752 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4e82ff91d18b4f17b819e366f95ef752; no voters:
I20260924 04:12:55.209600 22490 leader_election.cc:288] T 00000000000000000000000000000000 P 4e82ff91d18b4f17b819e366f95ef752 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:12:55.209664 22495 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e82ff91d18b4f17b819e366f95ef752 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:12:55.209861 22495 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4e82ff91d18b4f17b819e366f95ef752 [term 1 LEADER]: Becoming Leader. State: Replica: 4e82ff91d18b4f17b819e366f95ef752, State: Running, Role: LEADER
I20260924 04:12:55.210157 22495 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e82ff91d18b4f17b819e366f95ef752 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e82ff91d18b4f17b819e366f95ef752" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39241 } }
I20260924 04:12:55.210850 22490 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e82ff91d18b4f17b819e366f95ef752 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:12:55.211796 22497 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e82ff91d18b4f17b819e366f95ef752 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4e82ff91d18b4f17b819e366f95ef752" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e82ff91d18b4f17b819e366f95ef752" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39241 } } }
I20260924 04:12:55.211932 22497 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e82ff91d18b4f17b819e366f95ef752 [sys.catalog]: This master's current role is: LEADER
I20260924 04:12:55.211913 22496 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e82ff91d18b4f17b819e366f95ef752 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e82ff91d18b4f17b819e366f95ef752. Latest consensus state: current_term: 1 leader_uuid: "4e82ff91d18b4f17b819e366f95ef752" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e82ff91d18b4f17b819e366f95ef752" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39241 } } }
I20260924 04:12:55.211978 22496 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e82ff91d18b4f17b819e366f95ef752 [sys.catalog]: This master's current role is: LEADER
I20260924 04:12:55.212212 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:12:55.212373 22504 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:12:55.215828 22504 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:12:55.219841 22493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:55.214689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50715 (local address 127.0.98.62:39241)
0924 04:12:55.214802 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:55.214807 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:55.215199 (+ 392us) server_negotiation.cc:407] Connection header received
0924 04:12:55.215469 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:55.215472 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:55.215520 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:55.215573 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:12:55.216371 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:55.217136 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:55.218223 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:55.218462 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:55.218573 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:55.219403 (+ 830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:12:55.219406 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:12:55.219409 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:12:55.219421 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:12:55.219457 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:12:55.219467 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:12:55.219546 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:55.219634 (+ 88us) server_negotiation.cc:299] Negotiation successful
0924 04:12:55.219685 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 04:12:55.222167 22504 catalog_manager.cc:1347] Generated new cluster ID: 6bfa67917cac4adcbf436c1534a3e88e
I20260924 04:12:55.222230 22504 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:12:55.239948 22504 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:12:55.240991 22504 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:12:55.247661 22504 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4e82ff91d18b4f17b819e366f95ef752: Generated new TSK 0
I20260924 04:12:55.248318 22504 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:12:55.257865 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 22422
2026-09-24T04:12:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1403 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790222898807720-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:12:55 dist-test-slave-00ld krb5kdc[22521](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:12:55 dist-test-slave-00ld krb5kdc[22521](info): set up 2 sockets
Sep 24 04:12:55 dist-test-slave-00ld krb5kdc[22521](info): commencing operation
krb5kdc: starting...
W20260924 04:12:56.292976 392 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790222898807720-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790222898807720-392-0/krb5kdc/test-user.keytab.
Sep 24 04:12:56 dist-test-slave-00ld krb5kdc[22521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223176, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:12:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:12:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:12:56.415892 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790222898807720-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790222898807720-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790222898807720-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46231
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46231
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:12:56.523557 22534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:12:56.523792 22534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:12:56.523830 22534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:12:56.527678 22534 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:12:56.527738 22534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:12:56.527757 22534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:12:56.527773 22534 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:12:56.527788 22534 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:12:56.532824 22534 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790222898807720-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790222898807720-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46231
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46231
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22534
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790222898807720-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:12:56.533746 22534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:12:56.534665 22534 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:12:56.543087 22542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:12:56.543047 22540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:12:56.543156 22534 server_base.cc:1034] running on GCE node
W20260924 04:12:56.543337 22539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:12:56.544175 22534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:12:56.545334 22534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:12:56.546562 22534 hybrid_clock.cc:648] HybridClock initialized: now 1790223176546550 us; error 26 us; skew 500 ppm
Sep 24 04:12:56 dist-test-slave-00ld krb5kdc[22521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223176, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:12:56.549368 22534 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:12:56.550261 22534 webserver.cc:460] Webserver started at http://127.0.98.62:37901/ using document root <none> and password file <none>
I20260924 04:12:56.550768 22534 fs_manager.cc:359] Metadata directory not provided
I20260924 04:12:56.550825 22534 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:12:56.551074 22534 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:12:56.553129 22534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790222898807720-392-0/minicluster-data/master-0/data/instance:
uuid: "54afbcc3ebfa46bfbb30303fc08ee7ad"
format_stamp: "Formatted at 2026-09-24 04:12:56 on dist-test-slave-00ld"
server_key: "185145f05988bd782661e7c7d0059507"
server_key_iv: "74824707ff63a72842f601ee78c2fd11"
server_key_version: "encryptionkey@0"
I20260924 04:12:56.553633 22534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790222898807720-392-0/minicluster-data/master-0/wal/instance:
uuid: "54afbcc3ebfa46bfbb30303fc08ee7ad"
format_stamp: "Formatted at 2026-09-24 04:12:56 on dist-test-slave-00ld"
server_key: "185145f05988bd782661e7c7d0059507"
server_key_iv: "74824707ff63a72842f601ee78c2fd11"
server_key_version: "encryptionkey@0"
I20260924 04:12:56.557214 22534 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 04:12:56.559573 22549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:12:56.560838 22534 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:12:56.560973 22534 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790222898807720-392-0/minicluster-data/master-0/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790222898807720-392-0/minicluster-data/master-0/wal
uuid: "54afbcc3ebfa46bfbb30303fc08ee7ad"
format_stamp: "Formatted at 2026-09-24 04:12:56 on dist-test-slave-00ld"
server_key: "185145f05988bd782661e7c7d0059507"
server_key_iv: "74824707ff63a72842f601ee78c2fd11"
server_key_version: "encryptionkey@0"
I20260924 04:12:56.561146 22534 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790222898807720-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790222898807720-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790222898807720-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:12:56.565614 22534 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:12:56.568811 22534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:12:56.569005 22534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:12:56.575739 22534 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46231
I20260924 04:12:56.575817 22601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46231 every 8 connection(s)
I20260924 04:12:56.576736 22534 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:12:56.579473 22602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:12:56.581519 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 22534
I20260924 04:12:56.581660 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790222898807720-392-0/minicluster-data/master-0/wal/instance
I20260924 04:12:56.581904 392 external_mini_cluster.cc:1428] Setting key 327b6fda73a297520c4bcdedfa2fbf2d
I20260924 04:12:56.586187 22602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54afbcc3ebfa46bfbb30303fc08ee7ad: Bootstrap starting.
I20260924 04:12:56.587021 22605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:56.583251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54079 (local address 127.0.98.62:46231)
0924 04:12:56.583675 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:56.583691 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:56.583726 (+ 35us) server_negotiation.cc:407] Connection header received
0924 04:12:56.584340 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:56.584356 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:56.584630 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:56.584973 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:12:56.585821 (+ 848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:12:56.585826 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:12:56.585837 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:12:56.585848 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:12:56.585900 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:12:56.585919 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:12:56.586039 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:56.586154 (+ 115us) server_negotiation.cc:299] Negotiation successful
0924 04:12:56.586320 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":152,"threads_started":1}
I20260924 04:12:56.588405 22602 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 54afbcc3ebfa46bfbb30303fc08ee7ad: Neither blocks nor log segments found. Creating new log.
I20260924 04:12:56.589305 22602 log.cc:824] T 00000000000000000000000000000000 P 54afbcc3ebfa46bfbb30303fc08ee7ad: Log is configured to *not* fsync() on all Append() calls
I20260924 04:12:56.589850 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:12:56.593129 22602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54afbcc3ebfa46bfbb30303fc08ee7ad: No bootstrap required, opened a new log
I20260924 04:12:56.593489 22605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:56.591790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42077 (local address 127.0.98.62:46231)
0924 04:12:56.591917 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:56.591922 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:56.592113 (+ 191us) server_negotiation.cc:407] Connection header received
0924 04:12:56.592234 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:56.592236 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:56.592264 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:56.592329 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:12:56.593074 (+ 745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:12:56.593076 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:12:56.593078 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:12:56.593088 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:12:56.593113 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:12:56.593122 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:12:56.593220 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:56.593316 (+ 96us) server_negotiation.cc:299] Negotiation successful
0924 04:12:56.593348 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 04:12:56.596912 22602 raft_consensus.cc:348] T 00000000000000000000000000000000 P 54afbcc3ebfa46bfbb30303fc08ee7ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54afbcc3ebfa46bfbb30303fc08ee7ad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46231 } }
I20260924 04:12:56.597141 22602 raft_consensus.cc:374] T 00000000000000000000000000000000 P 54afbcc3ebfa46bfbb30303fc08ee7ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:12:56.597224 22602 raft_consensus.cc:729] T 00000000000000000000000000000000 P 54afbcc3ebfa46bfbb30303fc08ee7ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54afbcc3ebfa46bfbb30303fc08ee7ad, State: Initialized, Role: FOLLOWER
I20260924 04:12:56.597707 22602 consensus_queue.cc:260] T 00000000000000000000000000000000 P 54afbcc3ebfa46bfbb30303fc08ee7ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54afbcc3ebfa46bfbb30303fc08ee7ad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46231 } }
I20260924 04:12:56.597852 22602 raft_consensus.cc:388] T 00000000000000000000000000000000 P 54afbcc3ebfa46bfbb30303fc08ee7ad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:12:56.597918 22602 raft_consensus.cc:482] T 00000000000000000000000000000000 P 54afbcc3ebfa46bfbb30303fc08ee7ad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:12:56.598012 22602 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 54afbcc3ebfa46bfbb30303fc08ee7ad [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:12:56.599087 22602 raft_consensus.cc:504] T 00000000000000000000000000000000 P 54afbcc3ebfa46bfbb30303fc08ee7ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54afbcc3ebfa46bfbb30303fc08ee7ad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46231 } }
I20260924 04:12:56.599469 22602 leader_election.cc:302] T 00000000000000000000000000000000 P 54afbcc3ebfa46bfbb30303fc08ee7ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 54afbcc3ebfa46bfbb30303fc08ee7ad; no voters:
I20260924 04:12:56.599807 22602 leader_election.cc:288] T 00000000000000000000000000000000 P 54afbcc3ebfa46bfbb30303fc08ee7ad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:12:56.599901 22612 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 54afbcc3ebfa46bfbb30303fc08ee7ad [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:12:56.600167 22612 raft_consensus.cc:686] T 00000000000000000000000000000000 P 54afbcc3ebfa46bfbb30303fc08ee7ad [term 1 LEADER]: Becoming Leader. State: Replica: 54afbcc3ebfa46bfbb30303fc08ee7ad, State: Running, Role: LEADER
I20260924 04:12:56.600518 22612 consensus_queue.cc:237] T 00000000000000000000000000000000 P 54afbcc3ebfa46bfbb30303fc08ee7ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54afbcc3ebfa46bfbb30303fc08ee7ad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46231 } }
I20260924 04:12:56.601091 22602 sys_catalog.cc:564] T 00000000000000000000000000000000 P 54afbcc3ebfa46bfbb30303fc08ee7ad [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:12:56.602290 22614 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54afbcc3ebfa46bfbb30303fc08ee7ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 54afbcc3ebfa46bfbb30303fc08ee7ad. Latest consensus state: current_term: 1 leader_uuid: "54afbcc3ebfa46bfbb30303fc08ee7ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54afbcc3ebfa46bfbb30303fc08ee7ad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46231 } } }
I20260924 04:12:56.602289 22613 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54afbcc3ebfa46bfbb30303fc08ee7ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "54afbcc3ebfa46bfbb30303fc08ee7ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54afbcc3ebfa46bfbb30303fc08ee7ad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46231 } } }
I20260924 04:12:56.602499 22613 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54afbcc3ebfa46bfbb30303fc08ee7ad [sys.catalog]: This master's current role is: LEADER
I20260924 04:12:56.602499 22614 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54afbcc3ebfa46bfbb30303fc08ee7ad [sys.catalog]: This master's current role is: LEADER
I20260924 04:12:56.603447 22622 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:12:56.606186 22622 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:12:56.611773 22622 catalog_manager.cc:1347] Generated new cluster ID: b3360e4cb9134c1ca1561dad7a557ba0
I20260924 04:12:56.611841 22622 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:12:56.626832 22622 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:12:56.627171 22622 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:12:56.632896 22622 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 54afbcc3ebfa46bfbb30303fc08ee7ad: Generated new TSK 0
I20260924 04:12:56.633646 22622 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:12:56.639406 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 22534
2026-09-24T04:12:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1381 ms)
[----------] 18 tests from AuthTokenIssuingTest (25453 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222898807720-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:12:56 dist-test-slave-00ld krb5kdc[22633](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:12:56 dist-test-slave-00ld krb5kdc[22633](info): set up 2 sockets
Sep 24 04:12:56 dist-test-slave-00ld krb5kdc[22633](info): commencing operation
krb5kdc: starting...
W20260924 04:12:57.681084 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222898807720-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222898807720-392-0/krb5kdc/test-user.keytab.
Sep 24 04:12:57 dist-test-slave-00ld krb5kdc[22633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:12:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:12:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:12:57.803501 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222898807720-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222898807720-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222898807720-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33555
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33555,127.0.98.61:33925,127.0.98.60:42761
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:12:57.912583 22646 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 04:12:57.912878 22646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:12:57.912899 22646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:12:57.912936 22646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:12:57.916754 22646 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:12:57.916802 22646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:12:57.916821 22646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:12:57.916837 22646 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:12:57.916852 22646 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:12:57.921976 22646 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222898807720-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222898807720-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33555,127.0.98.61:33925,127.0.98.60:42761
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33555
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22646
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222898807720-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:12:57.923007 22646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:12:57.924016 22646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:12:57.932509 22651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:12:57.932533 22652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:12:57.932660 22646 server_base.cc:1034] running on GCE node
W20260924 04:12:57.932542 22654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:12:57.933425 22646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:12:57.934527 22646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:12:57.935709 22646 hybrid_clock.cc:648] HybridClock initialized: now 1790223177935693 us; error 28 us; skew 500 ppm
Sep 24 04:12:57 dist-test-slave-00ld krb5kdc[22633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223177, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:12:57.938531 22646 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:12:57.939474 22646 webserver.cc:460] Webserver started at http://127.0.98.62:45485/ using document root <none> and password file <none>
I20260924 04:12:57.940037 22646 fs_manager.cc:359] Metadata directory not provided
I20260924 04:12:57.940127 22646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:12:57.940361 22646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:12:57.942701 22646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222898807720-392-0/minicluster-data/master-0/data/instance:
uuid: "b4bf91c758b84f579814f360b1560d8c"
format_stamp: "Formatted at 2026-09-24 04:12:57 on dist-test-slave-00ld"
server_key: "690fd295fbd037fbdfc85129e575a322"
server_key_iv: "300c77f4b7d7e250db6c776c29cd87cf"
server_key_version: "encryptionkey@0"
I20260924 04:12:57.943322 22646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222898807720-392-0/minicluster-data/master-0/wal/instance:
uuid: "b4bf91c758b84f579814f360b1560d8c"
format_stamp: "Formatted at 2026-09-24 04:12:57 on dist-test-slave-00ld"
server_key: "690fd295fbd037fbdfc85129e575a322"
server_key_iv: "300c77f4b7d7e250db6c776c29cd87cf"
server_key_version: "encryptionkey@0"
I20260924 04:12:57.947103 22646 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260924 04:12:57.949656 22661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:12:57.950824 22646 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:12:57.950969 22646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222898807720-392-0/minicluster-data/master-0/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222898807720-392-0/minicluster-data/master-0/wal
uuid: "b4bf91c758b84f579814f360b1560d8c"
format_stamp: "Formatted at 2026-09-24 04:12:57 on dist-test-slave-00ld"
server_key: "690fd295fbd037fbdfc85129e575a322"
server_key_iv: "300c77f4b7d7e250db6c776c29cd87cf"
server_key_version: "encryptionkey@0"
I20260924 04:12:57.951134 22646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222898807720-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222898807720-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222898807720-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:12:57.971098 22646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:12:57.973989 22646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:12:57.974205 22646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:12:57.980875 22646 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33555
I20260924 04:12:57.981170 22713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33555 every 8 connection(s)
I20260924 04:12:57.981854 22646 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:12:57.984565 22714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:12:57.988231 22714 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33555 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:12:57.989727 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 22646
I20260924 04:12:57.989880 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222898807720-392-0/minicluster-data/master-0/wal/instance
I20260924 04:12:57.990188 392 external_mini_cluster.cc:1428] Setting key 4325f8bfd1fa1dd1f5e27b03cf5f8908
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Sep 24 04:12:57 dist-test-slave-00ld krb5kdc[22633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223177, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
I20260924 04:12:58.002825 22717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:57.990553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34378 (local address 127.0.98.62:33555)
0924 04:12:57.991117 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:57.991130 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:57.991176 (+ 46us) server_negotiation.cc:407] Connection header received
0924 04:12:57.992224 (+ 1048us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:57.992235 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:57.992411 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:57.992599 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:12:57.993459 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:57.994409 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:57.995368 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:57.995727 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:57.995892 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:57.998257 (+ 2365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:12:57.998262 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:12:57.998271 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:12:57.998291 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:12:57.999819 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:12:58.000358 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:12:58.000367 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:12:58.000372 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:12:58.000524 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:12:58.000910 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:12:58.000913 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:12:58.000914 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:12:58.001297 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:12:58.001414 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:58.001927 (+ 513us) server_negotiation.cc:299] Negotiation successful
0924 04:12:58.002192 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":420,"spinlock_wait_cycles":41856,"thread_start_us":162,"threads_started":1}
I20260924 04:12:58.002825 22716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:57.990553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33555 (local address 127.0.0.1:34378)
0924 04:12:57.991012 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:57.991064 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 04:12:57.991903 (+ 839us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:12:57.992598 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:12:57.992603 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:12:57.993080 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:12:57.993312 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:57.993324 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:57.994533 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:57.994536 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:57.995246 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:57.995254 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:57.995874 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:57.995878 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:57.996001 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:57.996637 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:12:57.996653 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:12:57.998108 (+ 1455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:12:57.999923 (+ 1815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:12:57.999934 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:12:57.999944 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:12:58.000218 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:12:58.000632 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:12:58.000635 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:12:58.000638 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:12:58.000807 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:12:58.001418 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:12:58.001427 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:12:58.001607 (+ 180us) client_negotiation.cc:769] Sending connection context
0924 04:12:58.001803 (+ 196us) client_negotiation.cc:240] Negotiation successful
0924 04:12:58.002051 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":135,"threads_started":1}
I20260924 04:12:58.005173 22714 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33925 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:12:58.005909 22716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:58.005681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33925 (local address 127.0.0.1:48810)
0924 04:12:58.005794 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:58.005857 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260924 04:12:58.006098 22665 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33925: connect: Connection refused (error 111)
I20260924 04:12:58.006868 22716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:58.006673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33925 (local address 127.0.0.1:48820)
0924 04:12:58.006776 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:58.006822 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260924 04:12:58.007020 22714 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:33925: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33925: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222898807720-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222898807720-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260924 04:12:58.016745 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222898807720-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222898807720-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222898807720-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222898807720-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:33925
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222898807720-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33555,127.0.98.61:33925,127.0.98.60:42761
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:12:58.056252 22714 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33925 } attempt: 1
I20260924 04:12:58.057159 22716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:58.056863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33925 (local address 127.0.0.1:48830)
0924 04:12:58.057009 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:58.057081 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260924 04:12:58.057889 22716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:58.057670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33925 (local address 127.0.0.1:48846)
0924 04:12:58.057774 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:58.057832 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260924 04:12:58.058022 22714 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:33925: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33925: connect: Connection refused (error 111)
W20260924 04:12:58.125880 22721 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 04:12:58.126209 22721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:12:58.126283 22721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
I20260924 04:12:58.126230 22714 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33925 } attempt: 2
W20260924 04:12:58.126364 22721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260924 04:12:58.127063 22716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:58.126768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33925 (local address 127.0.0.1:48852)
0924 04:12:58.126911 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:58.126988 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260924 04:12:58.127794 22716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:58.127573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33925 (local address 127.0.0.1:48866)
0924 04:12:58.127683 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:58.127732 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260924 04:12:58.127918 22714 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:33925: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33925: connect: Connection refused (error 111)
W20260924 04:12:58.130577 22721 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:12:58.130654 22721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:12:58.130692 22721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:12:58.130750 22721 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:12:58.130791 22721 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:12:58.135824 22721 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222898807720-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222898807720-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33555,127.0.98.61:33925,127.0.98.60:42761
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222898807720-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:33925
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222898807720-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22721
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222898807720-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:12:58.136919 22721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:12:58.137902 22721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:12:58.145905 22727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:12:58.145915 22729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:12:58.145919 22726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:12:58.145980 22721 server_base.cc:1034] running on GCE node
I20260924 04:12:58.146793 22721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:12:58.147814 22721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:12:58.149027 22721 hybrid_clock.cc:648] HybridClock initialized: now 1790223178149011 us; error 29 us; skew 500 ppm
Sep 24 04:12:58 dist-test-slave-00ld krb5kdc[22633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223178, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:12:58.151685 22721 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260924 04:12:58.152616 22721 webserver.cc:460] Webserver started at http://127.0.98.61:35137/ using document root <none> and password file <none>
I20260924 04:12:58.153213 22721 fs_manager.cc:359] Metadata directory not provided
I20260924 04:12:58.153306 22721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:12:58.153540 22721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:12:58.155630 22721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222898807720-392-0/minicluster-data/master-1/data/instance:
uuid: "523616d387004f04a984e8cb42ec8c78"
format_stamp: "Formatted at 2026-09-24 04:12:58 on dist-test-slave-00ld"
server_key: "1f6f322bed570a01c0f992f1619ca94a"
server_key_iv: "b868e9aa140d38b815e4c17b28cce22f"
server_key_version: "encryptionkey@0"
I20260924 04:12:58.156203 22721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222898807720-392-0/minicluster-data/master-1/wal/instance:
uuid: "523616d387004f04a984e8cb42ec8c78"
format_stamp: "Formatted at 2026-09-24 04:12:58 on dist-test-slave-00ld"
server_key: "1f6f322bed570a01c0f992f1619ca94a"
server_key_iv: "b868e9aa140d38b815e4c17b28cce22f"
server_key_version: "encryptionkey@0"
I20260924 04:12:58.160056 22721 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260924 04:12:58.162593 22736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:12:58.163820 22721 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 04:12:58.164017 22721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222898807720-392-0/minicluster-data/master-1/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222898807720-392-0/minicluster-data/master-1/wal
uuid: "523616d387004f04a984e8cb42ec8c78"
format_stamp: "Formatted at 2026-09-24 04:12:58 on dist-test-slave-00ld"
server_key: "1f6f322bed570a01c0f992f1619ca94a"
server_key_iv: "b868e9aa140d38b815e4c17b28cce22f"
server_key_version: "encryptionkey@0"
I20260924 04:12:58.164142 22721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222898807720-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222898807720-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222898807720-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:12:58.179826 22721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:12:58.183063 22721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:12:58.183264 22721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:12:58.190037 22721 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:33925
I20260924 04:12:58.190245 22788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:33925 every 8 connection(s)
I20260924 04:12:58.191013 22721 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222898807720-392-0/minicluster-data/master-1/data/info.pb
I20260924 04:12:58.192397 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 22721
I20260924 04:12:58.192536 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222898807720-392-0/minicluster-data/master-1/wal/instance
I20260924 04:12:58.192852 392 external_mini_cluster.cc:1428] Setting key 35451801c77d202bead3b8db4bb68360
I20260924 04:12:58.194249 22789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:12:58.198076 22789 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33555 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 24 04:12:58 dist-test-slave-00ld krb5kdc[22633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223178, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:12:58.212630 22717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:58.199934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34386 (local address 127.0.98.62:33555)
0924 04:12:58.200068 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:58.200072 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:58.201113 (+ 1041us) server_negotiation.cc:407] Connection header received
0924 04:12:58.202005 (+ 892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:58.202009 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:58.202067 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:58.202148 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:12:58.203399 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:58.204265 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:58.205556 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:58.205898 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:58.205987 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:58.208641 (+ 2654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:12:58.208644 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:12:58.208646 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:12:58.208671 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:12:58.210282 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:12:58.210887 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:12:58.210890 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:12:58.210891 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:12:58.211002 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:12:58.211421 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:12:58.211423 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:12:58.211424 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:12:58.211598 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:12:58.211670 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:58.212329 (+ 659us) server_negotiation.cc:299] Negotiation successful
0924 04:12:58.212437 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 04:12:58.213358 22792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:58.200433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33555 (local address 127.0.0.1:34386)
0924 04:12:58.200955 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:58.200996 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 04:12:58.201800 (+ 804us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:12:58.202303 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:12:58.202310 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:12:58.202699 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:12:58.203251 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:58.203266 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:58.204421 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:58.204424 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:58.205420 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:58.205430 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:58.206072 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:58.206075 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:58.206212 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:58.206819 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:12:58.206835 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:12:58.208441 (+ 1606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:12:58.210404 (+ 1963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:12:58.210411 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:12:58.210427 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:12:58.210788 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:12:58.211096 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:12:58.211099 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:12:58.211110 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:12:58.211318 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:12:58.211701 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:12:58.211708 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:12:58.212060 (+ 352us) client_negotiation.cc:769] Sending connection context
0924 04:12:58.212236 (+ 176us) client_negotiation.cc:240] Negotiation successful
0924 04:12:58.212472 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":167,"threads_started":1}
I20260924 04:12:58.214869 22789 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33925 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:12:58.219080 22714 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33925 } attempt: 3
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222898807720-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222898807720-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Sep 24 04:12:58 dist-test-slave-00ld krb5kdc[22633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223178, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 04:12:58.220638 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222898807720-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222898807720-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222898807720-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222898807720-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:42761
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222898807720-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33555,127.0.98.61:33925,127.0.98.60:42761
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 04:12:58 dist-test-slave-00ld krb5kdc[22633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223177, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 04:12:58.225342 22792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:58.215406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33925 (local address 127.0.0.1:48876)
0924 04:12:58.215541 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:58.215557 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:12:58.215638 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:12:58.216186 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:12:58.216188 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:12:58.216310 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:12:58.216390 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:58.216394 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:58.217520 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:58.217523 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:58.218055 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:58.218063 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:58.218737 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:58.218740 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:58.218810 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:58.219299 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:12:58.219312 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:12:58.221092 (+ 1780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:12:58.223496 (+ 2404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:12:58.223500 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:12:58.223502 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:12:58.223747 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:12:58.224053 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:12:58.224056 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:12:58.224058 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:12:58.224138 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:12:58.224705 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:12:58.224712 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:12:58.224825 (+ 113us) client_negotiation.cc:769] Sending connection context
0924 04:12:58.224946 (+ 121us) client_negotiation.cc:240] Negotiation successful
0924 04:12:58.225110 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260924 04:12:58.226122 22794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:58.215504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48876 (local address 127.0.98.61:33925)
0924 04:12:58.215841 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:58.215850 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:58.215869 (+ 19us) server_negotiation.cc:407] Connection header received
0924 04:12:58.215902 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:58.215905 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:58.216068 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:58.216206 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:12:58.216489 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:58.217380 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:58.218177 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:58.218625 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:58.218710 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:58.221236 (+ 2526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:12:58.221243 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:12:58.221254 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:12:58.221272 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:12:58.223378 (+ 2106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:12:58.223871 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:12:58.223877 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:12:58.223879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:12:58.223958 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:12:58.224223 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:12:58.224228 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:12:58.224230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:12:58.224553 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:12:58.224758 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:58.225648 (+ 890us) server_negotiation.cc:299] Negotiation successful
0924 04:12:58.225823 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":135,"threads_started":1}
I20260924 04:12:58.228201 22789 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42761 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:12:58.229000 22792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:58.228627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42761 (local address 127.0.0.1:35290)
0924 04:12:58.228812 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:58.228904 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
W20260924 04:12:58.229197 22739 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42761: connect: Connection refused (error 111)
I20260924 04:12:58.229488 22716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:58.219467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33925 (local address 127.0.0.1:48886)
0924 04:12:58.219649 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:58.219665 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:12:58.219744 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:12:58.220223 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:12:58.220226 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:12:58.220400 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:12:58.221037 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:58.221051 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:58.222189 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:58.222192 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:58.222717 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:58.222725 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:58.223186 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:58.223189 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:58.223291 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:58.223727 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:12:58.223746 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:12:58.225328 (+ 1582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:12:58.227878 (+ 2550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:12:58.227885 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:12:58.227888 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:12:58.228177 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:12:58.228467 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:12:58.228471 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:12:58.228476 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:12:58.228581 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:12:58.228987 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:12:58.228990 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:12:58.229094 (+ 104us) client_negotiation.cc:769] Sending connection context
0924 04:12:58.229198 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 04:12:58.229334 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99}
I20260924 04:12:58.229521 22795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:58.219532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48886 (local address 127.0.98.61:33925)
0924 04:12:58.219969 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:58.219978 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:58.219992 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:12:58.220029 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:58.220032 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:58.220073 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:58.220167 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:12:58.221182 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:58.222049 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:58.222832 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:58.223078 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:58.223183 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:58.225633 (+ 2450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:12:58.225636 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:12:58.225638 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:12:58.225657 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:12:58.227759 (+ 2102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:12:58.228281 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:12:58.228287 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:12:58.228289 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:12:58.228365 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:12:58.228708 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:12:58.228711 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:12:58.228713 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:12:58.228887 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:12:58.228994 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:58.229247 (+ 253us) server_negotiation.cc:299] Negotiation successful
0924 04:12:58.229349 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":103,"threads_started":1}
I20260924 04:12:58.230082 22714 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42761 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:12:58.230866 22716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:58.230543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42761 (local address 127.0.0.1:35300)
0924 04:12:58.230705 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:58.230769 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260924 04:12:58.231217 22792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:58.230612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42761 (local address 127.0.0.1:35316)
0924 04:12:58.231080 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:58.231139 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390}
W20260924 04:12:58.231375 22789 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42761: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42761: connect: Connection refused (error 111)
I20260924 04:12:58.231837 22716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:58.231302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42761 (local address 127.0.0.1:35330)
0924 04:12:58.231691 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:58.231750 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":307}
W20260924 04:12:58.232002 22714 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42761: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42761: connect: Connection refused (error 111)
I20260924 04:12:58.263183 22714 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42761 } attempt: 1
I20260924 04:12:58.264097 22716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:58.263757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42761 (local address 127.0.0.1:35338)
0924 04:12:58.263913 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:58.264014 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260924 04:12:58.265043 22716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:58.264807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42761 (local address 127.0.0.1:35340)
0924 04:12:58.264917 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:58.264974 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260924 04:12:58.265180 22714 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42761: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42761: connect: Connection refused (error 111)
I20260924 04:12:58.280652 22789 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42761 } attempt: 1
I20260924 04:12:58.281477 22792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:58.281170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42761 (local address 127.0.0.1:35352)
0924 04:12:58.281327 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:58.281394 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260924 04:12:58.282231 22792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:58.282012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42761 (local address 127.0.0.1:35368)
0924 04:12:58.282121 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:58.282172 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260924 04:12:58.282389 22789 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42761: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42761: connect: Connection refused (error 111)
W20260924 04:12:58.333145 22796 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 04:12:58.333438 22796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:12:58.333469 22796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:12:58.333554 22796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:12:58.337673 22796 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:12:58.337738 22796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:12:58.337756 22796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:12:58.337773 22796 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:12:58.337788 22796 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:12:58.340416 22714 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42761 } attempt: 2
I20260924 04:12:58.341306 22716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:58.340999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42761 (local address 127.0.0.1:35376)
0924 04:12:58.341147 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:58.341228 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260924 04:12:58.342330 22716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:58.342000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42761 (local address 127.0.0.1:35392)
0924 04:12:58.342190 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:58.342251 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260924 04:12:58.342505 22714 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42761: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42761: connect: Connection refused (error 111)
I20260924 04:12:58.343963 22796 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222898807720-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222898807720-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33555,127.0.98.61:33925,127.0.98.60:42761
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222898807720-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:42761
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222898807720-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22796
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222898807720-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:12:58.345130 22796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:12:58.346143 22796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:12:58.350548 22789 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42761 } attempt: 2
I20260924 04:12:58.351280 22792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:58.350994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42761 (local address 127.0.0.1:35408)
0924 04:12:58.351144 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:58.351206 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260924 04:12:58.351925 22792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:58.351724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42761 (local address 127.0.0.1:35410)
0924 04:12:58.351825 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:58.351873 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260924 04:12:58.352084 22789 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42761: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42761: connect: Connection refused (error 111)
W20260924 04:12:58.354933 22805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:12:58.354996 22803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:12:58.355160 22796 server_base.cc:1034] running on GCE node
W20260924 04:12:58.355017 22802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:12:58.355863 22796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:12:58.357081 22796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:12:58.358265 22796 hybrid_clock.cc:648] HybridClock initialized: now 1790223178358235 us; error 57 us; skew 500 ppm
Sep 24 04:12:58 dist-test-slave-00ld krb5kdc[22633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223178, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:12:58.361142 22796 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260924 04:12:58.362033 22796 webserver.cc:460] Webserver started at http://127.0.98.60:34855/ using document root <none> and password file <none>
I20260924 04:12:58.362639 22796 fs_manager.cc:359] Metadata directory not provided
I20260924 04:12:58.362740 22796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:12:58.362982 22796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:12:58.365180 22796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222898807720-392-0/minicluster-data/master-2/data/instance:
uuid: "f96150155eec4ab9bd4678293cb33541"
format_stamp: "Formatted at 2026-09-24 04:12:58 on dist-test-slave-00ld"
server_key: "1575ecf54e48dd888d440f6d010fa3f2"
server_key_iv: "f451c29a3d507cd2ef0b5d0ed1becf77"
server_key_version: "encryptionkey@0"
I20260924 04:12:58.365743 22796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222898807720-392-0/minicluster-data/master-2/wal/instance:
uuid: "f96150155eec4ab9bd4678293cb33541"
format_stamp: "Formatted at 2026-09-24 04:12:58 on dist-test-slave-00ld"
server_key: "1575ecf54e48dd888d440f6d010fa3f2"
server_key_iv: "f451c29a3d507cd2ef0b5d0ed1becf77"
server_key_version: "encryptionkey@0"
I20260924 04:12:58.369534 22796 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 04:12:58.371912 22812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:12:58.373172 22796 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 04:12:58.373310 22796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222898807720-392-0/minicluster-data/master-2/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222898807720-392-0/minicluster-data/master-2/wal
uuid: "f96150155eec4ab9bd4678293cb33541"
format_stamp: "Formatted at 2026-09-24 04:12:58 on dist-test-slave-00ld"
server_key: "1575ecf54e48dd888d440f6d010fa3f2"
server_key_iv: "f451c29a3d507cd2ef0b5d0ed1becf77"
server_key_version: "encryptionkey@0"
I20260924 04:12:58.373415 22796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222898807720-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222898807720-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222898807720-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:12:58.413861 22796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:12:58.417303 22796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:12:58.417515 22796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:12:58.424497 22796 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:42761
I20260924 04:12:58.424827 22864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:42761 every 8 connection(s)
I20260924 04:12:58.425559 22796 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222898807720-392-0/minicluster-data/master-2/data/info.pb
I20260924 04:12:58.428241 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 22796
I20260924 04:12:58.428386 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790222898807720-392-0/minicluster-data/master-2/wal/instance
I20260924 04:12:58.428390 22865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:12:58.428791 392 external_mini_cluster.cc:1428] Setting key 3f5fc6df6462f7a2a76e25472b2589d8
I20260924 04:12:58.433395 22865 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33555 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 04:12:58 dist-test-slave-00ld krb5kdc[22633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 24 04:12:58 dist-test-slave-00ld krb5kdc[22633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223178, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:12:58.441661 22714 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42761 } attempt: 3
I20260924 04:12:58.443284 22789 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42761 } attempt: 3
I20260924 04:12:58.444042 22868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:58.430411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35422 (local address 127.0.98.60:42761)
0924 04:12:58.430839 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:58.430855 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:58.430889 (+ 34us) server_negotiation.cc:407] Connection header received
0924 04:12:58.431553 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:58.431570 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:58.431895 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:58.432243 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:12:58.433124 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:58.434198 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:58.435257 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:58.435902 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:58.436076 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:58.438704 (+ 2628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:12:58.438709 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:12:58.438718 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:12:58.438769 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:12:58.440642 (+ 1873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:12:58.441114 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:12:58.441121 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:12:58.441128 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:12:58.441241 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:12:58.441591 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:12:58.441598 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:12:58.441601 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:12:58.442154 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:12:58.442364 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:58.442669 (+ 305us) server_negotiation.cc:299] Negotiation successful
0924 04:12:58.442956 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":122,"threads_started":1}
I20260924 04:12:58.445013 22717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:58.435214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34402 (local address 127.0.98.62:33555)
0924 04:12:58.435345 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:58.435349 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:58.435625 (+ 276us) server_negotiation.cc:407] Connection header received
0924 04:12:58.435768 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:58.435771 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:58.435820 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:58.435948 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:12:58.436865 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:58.437339 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:58.438321 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:58.438688 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:58.438776 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:58.441177 (+ 2401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:12:58.441180 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:12:58.441182 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:12:58.441210 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:12:58.443072 (+ 1862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:12:58.443569 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:12:58.443573 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:12:58.443575 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:12:58.443663 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:12:58.443952 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:12:58.443955 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:12:58.443957 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:12:58.444198 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:12:58.444273 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:58.444739 (+ 466us) server_negotiation.cc:299] Negotiation successful
0924 04:12:58.444858 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 04:12:58.445515 22869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:58.435118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33555 (local address 127.0.0.1:34402)
0924 04:12:58.435474 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:58.435542 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0924 04:12:58.435661 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:12:58.436013 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:12:58.436018 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:12:58.436479 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:12:58.436723 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:58.436733 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:58.437465 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:58.437471 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:58.438190 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:58.438201 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:58.438796 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:58.438802 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:58.438933 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:58.439442 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:12:58.439462 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:12:58.441023 (+ 1561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:12:58.443177 (+ 2154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:12:58.443184 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:12:58.443190 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:12:58.443446 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:12:58.443772 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:12:58.443775 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:12:58.443777 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:12:58.443858 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:12:58.444353 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:12:58.444359 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:12:58.444476 (+ 117us) client_negotiation.cc:769] Sending connection context
0924 04:12:58.445136 (+ 660us) client_negotiation.cc:240] Negotiation successful
0924 04:12:58.445337 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":141,"threads_started":1}
I20260924 04:12:58.447794 22865 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33925 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 04:12:58 dist-test-slave-00ld krb5kdc[22633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223177, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 24 04:12:58 dist-test-slave-00ld krb5kdc[22633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223178, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 04:12:58.453656 22716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:58.442337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42761 (local address 127.0.0.1:35428)
0924 04:12:58.442533 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:58.442549 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 04:12:58.442641 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:12:58.443576 (+ 935us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:12:58.443582 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:12:58.443753 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:12:58.443866 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:58.443874 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:58.444978 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:58.444981 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:58.445809 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:58.445820 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:58.446649 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:58.446653 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:58.446730 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:58.447718 (+ 988us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:12:58.447732 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:12:58.449913 (+ 2181us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:12:58.451968 (+ 2055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:12:58.451971 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:12:58.451973 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:12:58.452224 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:12:58.452559 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:12:58.452562 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:12:58.452564 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:12:58.452661 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:12:58.453135 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:12:58.453139 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:12:58.453241 (+ 102us) client_negotiation.cc:769] Sending connection context
0924 04:12:58.453354 (+ 113us) client_negotiation.cc:240] Negotiation successful
0924 04:12:58.453480 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116}
I20260924 04:12:58.453665 22870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:58.442222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35428 (local address 127.0.98.60:42761)
0924 04:12:58.443334 (+ 1112us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:58.443344 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:58.443358 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:12:58.443391 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:58.443395 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:58.443434 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:58.443980 (+ 546us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:12:58.444073 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:58.444858 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:58.446163 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:58.446510 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:58.446633 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:58.450065 (+ 3432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:12:58.450069 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:12:58.450071 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:12:58.450094 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:12:58.451845 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:12:58.452335 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:12:58.452338 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:12:58.452340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:12:58.452439 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:12:58.452800 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:12:58.452804 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:12:58.452805 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:12:58.453017 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:12:58.453135 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:58.453404 (+ 269us) server_negotiation.cc:299] Negotiation successful
0924 04:12:58.453521 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1029,"thread_start_us":116,"threads_started":1}
Sep 24 04:12:58 dist-test-slave-00ld krb5kdc[22633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 24 04:12:58 dist-test-slave-00ld krb5kdc[22633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223178, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 04:12:58.457127 22792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:58.443705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42761 (local address 127.0.0.1:35440)
0924 04:12:58.443967 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:58.443980 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:12:58.444050 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:12:58.444654 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:12:58.444657 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:12:58.444870 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:12:58.444971 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:58.444980 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:58.445976 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:58.445980 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:58.446513 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:58.446522 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:58.447429 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:58.447433 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:58.447532 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:58.448003 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:12:58.448018 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:12:58.450951 (+ 2933us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:12:58.455084 (+ 4133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:12:58.455088 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:12:58.455093 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:12:58.455335 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:12:58.455647 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:12:58.455651 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:12:58.455654 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:12:58.455763 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:12:58.456551 (+ 788us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:12:58.456558 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:12:58.456669 (+ 111us) client_negotiation.cc:769] Sending connection context
0924 04:12:58.456786 (+ 117us) client_negotiation.cc:240] Negotiation successful
0924 04:12:58.456923 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174}
I20260924 04:12:58.457772 22868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:58.444180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35440 (local address 127.0.98.60:42761)
0924 04:12:58.444389 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:58.444393 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:58.444403 (+ 10us) server_negotiation.cc:407] Connection header received
0924 04:12:58.444430 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:58.444434 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:58.444488 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:58.445092 (+ 604us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:12:58.445176 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:58.445851 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:58.446934 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:58.447297 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:58.447425 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:58.451072 (+ 3647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:12:58.451076 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:12:58.451079 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:12:58.451135 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:12:58.454963 (+ 3828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:12:58.455467 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:12:58.455483 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:12:58.455485 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:12:58.455561 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:12:58.456199 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:12:58.456203 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:12:58.456206 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:12:58.456426 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:12:58.457412 (+ 986us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:58.457486 (+ 74us) server_negotiation.cc:299] Negotiation successful
0924 04:12:58.457608 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":154,"thread_start_us":89,"threads_started":1}
I20260924 04:12:58.458866 22714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4bf91c758b84f579814f360b1560d8c: Bootstrap starting.
I20260924 04:12:58.459429 22717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:58.447992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34406 (local address 127.0.98.62:33555)
0924 04:12:58.448086 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:58.448090 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:58.448447 (+ 357us) server_negotiation.cc:407] Connection header received
0924 04:12:58.448483 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:58.448488 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:58.448529 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:58.448607 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:12:58.449453 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:58.451301 (+ 1848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:58.452060 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:58.452289 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:58.452392 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:58.455273 (+ 2881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:12:58.455276 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:12:58.455279 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:12:58.455303 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:12:58.457106 (+ 1803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:12:58.457618 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:12:58.457623 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:12:58.457624 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:12:58.457711 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:12:58.458035 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:12:58.458038 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:12:58.458040 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:12:58.458260 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:12:58.458456 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:58.459135 (+ 679us) server_negotiation.cc:299] Negotiation successful
0924 04:12:58.459251 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 04:12:58.459728 22795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:58.448717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48890 (local address 127.0.98.61:33925)
0924 04:12:58.448831 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:58.448837 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:58.448856 (+ 19us) server_negotiation.cc:407] Connection header received
0924 04:12:58.448890 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:58.448895 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:58.448944 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:58.449024 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:12:58.449516 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:58.450292 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:58.452115 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:58.452493 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:58.452564 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:58.456122 (+ 3558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:12:58.456125 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:12:58.456127 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:12:58.456156 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:12:58.457809 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:12:58.458270 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:12:58.458273 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:12:58.458276 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:12:58.458358 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:12:58.458698 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:12:58.458704 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:12:58.458709 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:12:58.458975 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:12:58.459055 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:58.459420 (+ 365us) server_negotiation.cc:299] Negotiation successful
0924 04:12:58.459578 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 04:12:58.460142 22869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:58.448474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33925 (local address 127.0.0.1:48890)
0924 04:12:58.448588 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:58.448602 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:12:58.448670 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:12:58.449071 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:12:58.449075 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:12:58.449244 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:12:58.449378 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:58.449387 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:58.451114 (+ 1727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:58.451119 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:58.451993 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:58.452001 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:58.452727 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:58.452731 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:58.452833 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:58.453935 (+ 1102us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:12:58.453953 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:12:58.455988 (+ 2035us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:12:58.457920 (+ 1932us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:12:58.457923 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:12:58.457924 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:12:58.458158 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:12:58.458493 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:12:58.458496 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:12:58.458498 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:12:58.458586 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:12:58.459150 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:12:58.459153 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:12:58.459243 (+ 90us) client_negotiation.cc:769] Sending connection context
0924 04:12:58.459839 (+ 596us) client_negotiation.cc:240] Negotiation successful
0924 04:12:58.460000 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 04:12:58.460779 22789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 523616d387004f04a984e8cb42ec8c78: Bootstrap starting.
I20260924 04:12:58.460826 22865 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42761 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:12:58.463300 22789 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 523616d387004f04a984e8cb42ec8c78: Neither blocks nor log segments found. Creating new log.
I20260924 04:12:58.463300 22714 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b4bf91c758b84f579814f360b1560d8c: Neither blocks nor log segments found. Creating new log.
I20260924 04:12:58.464223 22714 log.cc:824] T 00000000000000000000000000000000 P b4bf91c758b84f579814f360b1560d8c: Log is configured to *not* fsync() on all Append() calls
I20260924 04:12:58.464222 22789 log.cc:824] T 00000000000000000000000000000000 P 523616d387004f04a984e8cb42ec8c78: Log is configured to *not* fsync() on all Append() calls
I20260924 04:12:58.466507 22789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 523616d387004f04a984e8cb42ec8c78: No bootstrap required, opened a new log
Sep 24 04:12:58 dist-test-slave-00ld krb5kdc[22633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223178, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 04:12:58.467119 22714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4bf91c758b84f579814f360b1560d8c: No bootstrap required, opened a new log
Sep 24 04:12:58 dist-test-slave-00ld krb5kdc[22633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 04:12:58.470976 22789 raft_consensus.cc:348] T 00000000000000000000000000000000 P 523616d387004f04a984e8cb42ec8c78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4bf91c758b84f579814f360b1560d8c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33555 } } peers { permanent_uuid: "523616d387004f04a984e8cb42ec8c78" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33925 } } peers { permanent_uuid: "f96150155eec4ab9bd4678293cb33541" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42761 } }
I20260924 04:12:58.471226 22789 raft_consensus.cc:729] T 00000000000000000000000000000000 P 523616d387004f04a984e8cb42ec8c78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 523616d387004f04a984e8cb42ec8c78, State: Initialized, Role: FOLLOWER
I20260924 04:12:58.471597 22870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:58.461333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35454 (local address 127.0.98.60:42761)
0924 04:12:58.461425 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:58.461428 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:58.461663 (+ 235us) server_negotiation.cc:407] Connection header received
0924 04:12:58.461792 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:58.461795 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:58.461837 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:58.461882 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:12:58.462395 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:58.463193 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:58.464295 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:58.464659 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:58.464749 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:58.467780 (+ 3031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:12:58.467787 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:12:58.467793 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:12:58.467822 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:12:58.469823 (+ 2001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:12:58.470313 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:12:58.470316 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:12:58.470318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:12:58.470393 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:12:58.470679 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:12:58.470685 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:12:58.470690 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:12:58.470896 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:12:58.470966 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:58.471297 (+ 331us) server_negotiation.cc:299] Negotiation successful
0924 04:12:58.471429 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 04:12:58.471755 22795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:58.460782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48904 (local address 127.0.98.61:33925)
0924 04:12:58.460885 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:58.460889 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:58.461057 (+ 168us) server_negotiation.cc:407] Connection header received
0924 04:12:58.461537 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:58.461540 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:58.461583 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:58.461653 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:12:58.462528 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:58.463238 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:58.464301 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:58.464645 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:58.464755 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:58.467836 (+ 3081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:12:58.467840 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:12:58.467842 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:12:58.467868 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:12:58.469823 (+ 1955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:12:58.470320 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:12:58.470324 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:12:58.470325 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:12:58.470394 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:12:58.470711 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:12:58.470718 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:12:58.470721 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:12:58.470897 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:12:58.471070 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:58.471428 (+ 358us) server_negotiation.cc:299] Negotiation successful
0924 04:12:58.471543 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 04:12:58.471570 22789 consensus_queue.cc:260] T 00000000000000000000000000000000 P 523616d387004f04a984e8cb42ec8c78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4bf91c758b84f579814f360b1560d8c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33555 } } peers { permanent_uuid: "523616d387004f04a984e8cb42ec8c78" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33925 } } peers { permanent_uuid: "f96150155eec4ab9bd4678293cb33541" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42761 } }
I20260924 04:12:58.472082 22869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:58.461309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42761 (local address 127.0.0.1:35454)
0924 04:12:58.461609 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0924 04:12:58.461624 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:12:58.461707 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:12:58.462022 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:12:58.462025 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:12:58.462194 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:12:58.462285 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:58.462292 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:58.463315 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:58.463320 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:58.464170 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:12:58.464179 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:12:58.464882 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:12:58.464885 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:12:58.464967 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:58.465516 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:12:58.465527 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:12:58.467633 (+ 2106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:12:58.469939 (+ 2306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:12:58.469942 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:12:58.469944 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:12:58.470209 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:12:58.470495 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:12:58.470498 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:12:58.470502 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:12:58.470584 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:12:58.471044 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:12:58.471047 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:12:58.471141 (+ 94us) client_negotiation.cc:769] Sending connection context
0924 04:12:58.471735 (+ 594us) client_negotiation.cc:240] Negotiation successful
0924 04:12:58.471863 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224}
I20260924 04:12:58.472442 22714 raft_consensus.cc:348] T 00000000000000000000000000000000 P b4bf91c758b84f579814f360b1560d8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4bf91c758b84f579814f360b1560d8c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33555 } } peers { permanent_uuid: "523616d387004f04a984e8cb42ec8c78" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33925 } } peers { permanent_uuid: "f96150155eec4ab9bd4678293cb33541" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42761 } }
I20260924 04:12:58.472754 22714 raft_consensus.cc:729] T 00000000000000000000000000000000 P b4bf91c758b84f579814f360b1560d8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4bf91c758b84f579814f360b1560d8c, State: Initialized, Role: FOLLOWER
I20260924 04:12:58.472946 22874 sys_catalog.cc:455] T 00000000000000000000000000000000 P 523616d387004f04a984e8cb42ec8c78 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4bf91c758b84f579814f360b1560d8c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33555 } } peers { permanent_uuid: "523616d387004f04a984e8cb42ec8c78" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33925 } } peers { permanent_uuid: "f96150155eec4ab9bd4678293cb33541" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42761 } } }
I20260924 04:12:58.473155 22874 sys_catalog.cc:458] T 00000000000000000000000000000000 P 523616d387004f04a984e8cb42ec8c78 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:12:58.473240 22714 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4bf91c758b84f579814f360b1560d8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4bf91c758b84f579814f360b1560d8c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33555 } } peers { permanent_uuid: "523616d387004f04a984e8cb42ec8c78" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33925 } } peers { permanent_uuid: "f96150155eec4ab9bd4678293cb33541" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42761 } }
I20260924 04:12:58.473793 22875 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4bf91c758b84f579814f360b1560d8c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4bf91c758b84f579814f360b1560d8c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33555 } } peers { permanent_uuid: "523616d387004f04a984e8cb42ec8c78" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33925 } } peers { permanent_uuid: "f96150155eec4ab9bd4678293cb33541" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42761 } } }
I20260924 04:12:58.473951 22875 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4bf91c758b84f579814f360b1560d8c [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:12:58.473959 22789 sys_catalog.cc:564] T 00000000000000000000000000000000 P 523616d387004f04a984e8cb42ec8c78 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:12:58.474722 22714 sys_catalog.cc:564] T 00000000000000000000000000000000 P b4bf91c758b84f579814f360b1560d8c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:12:58.475576 22865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f96150155eec4ab9bd4678293cb33541: Bootstrap starting.
W20260924 04:12:58.475708 22667 tablet.cc:2369] T 00000000000000000000000000000000 P b4bf91c758b84f579814f360b1560d8c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:12:58.479666 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260924 04:12:58.479910 22893 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 523616d387004f04a984e8cb42ec8c78: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:12:58.479988 22893 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260924 04:12:58.480243 22897 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b4bf91c758b84f579814f360b1560d8c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:12:58.480310 22897 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:12:58.481566 22688 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "b4bf91c758b84f579814f360b1560d8c"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34406
I20260924 04:12:58.481819 22688 raft_consensus.cc:482] T 00000000000000000000000000000000 P b4bf91c758b84f579814f360b1560d8c [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 04:12:58.481927 22688 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b4bf91c758b84f579814f360b1560d8c [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:12:58.483044 22865 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f96150155eec4ab9bd4678293cb33541: Neither blocks nor log segments found. Creating new log.
I20260924 04:12:58.483642 22688 raft_consensus.cc:504] T 00000000000000000000000000000000 P b4bf91c758b84f579814f360b1560d8c [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4bf91c758b84f579814f360b1560d8c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33555 } } peers { permanent_uuid: "523616d387004f04a984e8cb42ec8c78" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33925 } } peers { permanent_uuid: "f96150155eec4ab9bd4678293cb33541" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42761 } }
I20260924 04:12:58.484918 22763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b4bf91c758b84f579814f360b1560d8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "523616d387004f04a984e8cb42ec8c78"
I20260924 04:12:58.485136 22763 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 523616d387004f04a984e8cb42ec8c78 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:12:58.485158 22688 leader_election.cc:288] T 00000000000000000000000000000000 P b4bf91c758b84f579814f360b1560d8c [CANDIDATE]: Term 1 election: Requested vote from peers 523616d387004f04a984e8cb42ec8c78 (127.0.98.61:33925), f96150155eec4ab9bd4678293cb33541 (127.0.98.60:42761)
I20260924 04:12:58.482466 22721 master_runner.cc:186] Error getting master registration for 127.0.98.60:42761: OK, instance_id { permanent_uuid: "f96150155eec4ab9bd4678293cb33541" instance_seqno: 1790223178422489 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 04:12:58.485978 22721 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 04:12:58.486395 22763 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 523616d387004f04a984e8cb42ec8c78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4bf91c758b84f579814f360b1560d8c in term 1.
I20260924 04:12:58.483539 22646 master_runner.cc:186] Error getting master registration for 127.0.98.60:42761: OK, instance_id { permanent_uuid: "f96150155eec4ab9bd4678293cb33541" instance_seqno: 1790223178422489 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 04:12:58.486574 22646 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 04:12:58.486766 22665 leader_election.cc:302] T 00000000000000000000000000000000 P b4bf91c758b84f579814f360b1560d8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 523616d387004f04a984e8cb42ec8c78, b4bf91c758b84f579814f360b1560d8c; no voters:
I20260924 04:12:58.487035 22875 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b4bf91c758b84f579814f360b1560d8c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:12:58.487232 22875 raft_consensus.cc:686] T 00000000000000000000000000000000 P b4bf91c758b84f579814f360b1560d8c [term 1 LEADER]: Becoming Leader. State: Replica: b4bf91c758b84f579814f360b1560d8c, State: Running, Role: LEADER
I20260924 04:12:58.487284 22865 log.cc:824] T 00000000000000000000000000000000 P f96150155eec4ab9bd4678293cb33541: Log is configured to *not* fsync() on all Append() calls
I20260924 04:12:58.487553 22875 consensus_queue.cc:237] T 00000000000000000000000000000000 P b4bf91c758b84f579814f360b1560d8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4bf91c758b84f579814f360b1560d8c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33555 } } peers { permanent_uuid: "523616d387004f04a984e8cb42ec8c78" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33925 } } peers { permanent_uuid: "f96150155eec4ab9bd4678293cb33541" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42761 } }
I20260924 04:12:58.489447 22875 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4bf91c758b84f579814f360b1560d8c [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4bf91c758b84f579814f360b1560d8c. Latest consensus state: current_term: 1 leader_uuid: "b4bf91c758b84f579814f360b1560d8c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4bf91c758b84f579814f360b1560d8c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33555 } } peers { permanent_uuid: "523616d387004f04a984e8cb42ec8c78" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33925 } } peers { permanent_uuid: "f96150155eec4ab9bd4678293cb33541" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42761 } } }
I20260924 04:12:58.489580 22875 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4bf91c758b84f579814f360b1560d8c [sys.catalog]: This master's current role is: LEADER
I20260924 04:12:58.489784 22865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f96150155eec4ab9bd4678293cb33541: No bootstrap required, opened a new log
I20260924 04:12:58.489952 22910 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:12:58.490883 22910 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:12:58.493839 22865 raft_consensus.cc:348] T 00000000000000000000000000000000 P f96150155eec4ab9bd4678293cb33541 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4bf91c758b84f579814f360b1560d8c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33555 } } peers { permanent_uuid: "523616d387004f04a984e8cb42ec8c78" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33925 } } peers { permanent_uuid: "f96150155eec4ab9bd4678293cb33541" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42761 } }
I20260924 04:12:58.494093 22865 raft_consensus.cc:729] T 00000000000000000000000000000000 P f96150155eec4ab9bd4678293cb33541 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f96150155eec4ab9bd4678293cb33541, State: Initialized, Role: FOLLOWER
I20260924 04:12:58.494424 22865 consensus_queue.cc:260] T 00000000000000000000000000000000 P f96150155eec4ab9bd4678293cb33541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4bf91c758b84f579814f360b1560d8c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33555 } } peers { permanent_uuid: "523616d387004f04a984e8cb42ec8c78" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33925 } } peers { permanent_uuid: "f96150155eec4ab9bd4678293cb33541" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42761 } }
I20260924 04:12:58.495684 22865 sys_catalog.cc:564] T 00000000000000000000000000000000 P f96150155eec4ab9bd4678293cb33541 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:12:58.495985 22839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b4bf91c758b84f579814f360b1560d8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f96150155eec4ab9bd4678293cb33541"
I20260924 04:12:58.496227 22839 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f96150155eec4ab9bd4678293cb33541 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:12:58.497654 22839 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f96150155eec4ab9bd4678293cb33541 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4bf91c758b84f579814f360b1560d8c in term 1.
I20260924 04:12:58.498140 22838 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f96150155eec4ab9bd4678293cb33541 [term 1 FOLLOWER]: Refusing update from remote peer b4bf91c758b84f579814f360b1560d8c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:12:58.498431 22763 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 523616d387004f04a984e8cb42ec8c78 [term 1 FOLLOWER]: Refusing update from remote peer b4bf91c758b84f579814f360b1560d8c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:12:58.498953 22875 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b4bf91c758b84f579814f360b1560d8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f96150155eec4ab9bd4678293cb33541" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:12:58.499383 22875 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b4bf91c758b84f579814f360b1560d8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "523616d387004f04a984e8cb42ec8c78" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:12:58.505584 22870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:58.488027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35456 (local address 127.0.98.60:42761)
0924 04:12:58.488150 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:58.488154 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:58.488805 (+ 651us) server_negotiation.cc:407] Connection header received
0924 04:12:58.490143 (+ 1338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:58.490146 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:58.490190 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:58.490243 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:12:58.493220 (+ 2977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:58.493948 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:58.496079 (+ 2131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:58.496394 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:58.496490 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:58.500100 (+ 3610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:12:58.500104 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:12:58.500106 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:12:58.500135 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:12:58.501877 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:12:58.503775 (+ 1898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:12:58.503778 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:12:58.503780 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:12:58.503864 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:12:58.504199 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:12:58.504202 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:12:58.504204 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:12:58.504419 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:12:58.504552 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:58.505308 (+ 756us) server_negotiation.cc:299] Negotiation successful
0924 04:12:58.505425 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 04:12:58.507366 22874 sys_catalog.cc:455] T 00000000000000000000000000000000 P 523616d387004f04a984e8cb42ec8c78 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4bf91c758b84f579814f360b1560d8c. Latest consensus state: current_term: 1 leader_uuid: "b4bf91c758b84f579814f360b1560d8c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4bf91c758b84f579814f360b1560d8c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33555 } } peers { permanent_uuid: "523616d387004f04a984e8cb42ec8c78" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33925 } } peers { permanent_uuid: "f96150155eec4ab9bd4678293cb33541" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42761 } } }
I20260924 04:12:58.507525 22874 sys_catalog.cc:458] T 00000000000000000000000000000000 P 523616d387004f04a984e8cb42ec8c78 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:12:58.507829 22912 sys_catalog.cc:455] T 00000000000000000000000000000000 P f96150155eec4ab9bd4678293cb33541 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b4bf91c758b84f579814f360b1560d8c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4bf91c758b84f579814f360b1560d8c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33555 } } peers { permanent_uuid: "523616d387004f04a984e8cb42ec8c78" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33925 } } peers { permanent_uuid: "f96150155eec4ab9bd4678293cb33541" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42761 } } }
I20260924 04:12:58.507963 22912 sys_catalog.cc:458] T 00000000000000000000000000000000 P f96150155eec4ab9bd4678293cb33541 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:12:58.508083 22912 sys_catalog.cc:455] T 00000000000000000000000000000000 P f96150155eec4ab9bd4678293cb33541 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4bf91c758b84f579814f360b1560d8c. Latest consensus state: current_term: 1 leader_uuid: "b4bf91c758b84f579814f360b1560d8c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4bf91c758b84f579814f360b1560d8c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33555 } } peers { permanent_uuid: "523616d387004f04a984e8cb42ec8c78" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33925 } } peers { permanent_uuid: "f96150155eec4ab9bd4678293cb33541" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42761 } } }
I20260924 04:12:58.508162 22912 sys_catalog.cc:458] T 00000000000000000000000000000000 P f96150155eec4ab9bd4678293cb33541 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:12:58.509248 22795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:58.487890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48910 (local address 127.0.98.61:33925)
0924 04:12:58.487998 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:58.488002 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:58.497168 (+ 9166us) server_negotiation.cc:407] Connection header received
0924 04:12:58.497356 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:58.497360 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:58.497414 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:58.497470 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:12:58.498456 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:58.499234 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:58.500382 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:58.500766 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:58.500843 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:58.502559 (+ 1716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:12:58.502563 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:12:58.502567 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:12:58.502604 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:12:58.505933 (+ 3329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:12:58.506437 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:12:58.506441 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:12:58.506443 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:12:58.506526 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:12:58.507158 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:12:58.507162 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:12:58.507164 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:12:58.507379 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:12:58.507501 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:58.508977 (+ 1476us) server_negotiation.cc:299] Negotiation successful
0924 04:12:58.509094 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 04:12:58.511015 22717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:58.487337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34408 (local address 127.0.98.62:33555)
0924 04:12:58.487451 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:58.487458 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:58.488438 (+ 980us) server_negotiation.cc:407] Connection header received
0924 04:12:58.488887 (+ 449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:58.488890 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:58.488946 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:58.489001 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:12:58.491237 (+ 2236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:58.492004 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:58.495072 (+ 3068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:58.495405 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:58.495486 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:58.503347 (+ 7861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:12:58.503350 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:12:58.503353 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:12:58.503380 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:12:58.505128 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:12:58.507849 (+ 2721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:12:58.507854 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:12:58.507856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:12:58.507947 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:12:58.508261 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:12:58.508265 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:12:58.508267 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:12:58.508488 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:12:58.508621 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:58.510740 (+ 2119us) server_negotiation.cc:299] Negotiation successful
0924 04:12:58.510864 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 04:12:58.514380 22875 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4bf91c758b84f579814f360b1560d8c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b4bf91c758b84f579814f360b1560d8c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4bf91c758b84f579814f360b1560d8c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33555 } } peers { permanent_uuid: "523616d387004f04a984e8cb42ec8c78" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33925 } } peers { permanent_uuid: "f96150155eec4ab9bd4678293cb33541" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42761 } } }
I20260924 04:12:58.514508 22875 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4bf91c758b84f579814f360b1560d8c [sys.catalog]: This master's current role is: LEADER
I20260924 04:12:58.514767 22911 mvcc.cc:205] Tried to move back new op lower bound from 7332754139118166016 to 7332754139088703488. Current Snapshot: MvccSnapshot[applied={T|T < 7332754139118166016}]
I20260924 04:12:58.516044 22874 sys_catalog.cc:455] T 00000000000000000000000000000000 P 523616d387004f04a984e8cb42ec8c78 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b4bf91c758b84f579814f360b1560d8c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4bf91c758b84f579814f360b1560d8c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33555 } } peers { permanent_uuid: "523616d387004f04a984e8cb42ec8c78" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33925 } } peers { permanent_uuid: "f96150155eec4ab9bd4678293cb33541" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42761 } } }
I20260924 04:12:58.516158 22874 sys_catalog.cc:458] T 00000000000000000000000000000000 P 523616d387004f04a984e8cb42ec8c78 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:12:58.517091 22875 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4bf91c758b84f579814f360b1560d8c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b4bf91c758b84f579814f360b1560d8c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4bf91c758b84f579814f360b1560d8c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33555 } } peers { permanent_uuid: "523616d387004f04a984e8cb42ec8c78" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33925 } } peers { permanent_uuid: "f96150155eec4ab9bd4678293cb33541" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42761 } } }
I20260924 04:12:58.517200 22875 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4bf91c758b84f579814f360b1560d8c [sys.catalog]: This master's current role is: LEADER
I20260924 04:12:58.518332 22921 sys_catalog.cc:455] T 00000000000000000000000000000000 P f96150155eec4ab9bd4678293cb33541 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b4bf91c758b84f579814f360b1560d8c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4bf91c758b84f579814f360b1560d8c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33555 } } peers { permanent_uuid: "523616d387004f04a984e8cb42ec8c78" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33925 } } peers { permanent_uuid: "f96150155eec4ab9bd4678293cb33541" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42761 } } }
I20260924 04:12:58.518451 22921 sys_catalog.cc:458] T 00000000000000000000000000000000 P f96150155eec4ab9bd4678293cb33541 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:12:58.518564 22922 mvcc.cc:205] Tried to move back new op lower bound from 7332754139118166016 to 7332754139088703488. Current Snapshot: MvccSnapshot[applied={T|T < 7332754139118166016}]
W20260924 04:12:58.519359 22928 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f96150155eec4ab9bd4678293cb33541: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:12:58.519438 22928 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:12:58.526254 22910 catalog_manager.cc:1347] Generated new cluster ID: b52af584698c40b593370c331815e1d9
I20260924 04:12:58.526326 22910 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:12:58.543854 22910 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:12:58.545156 22910 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:12:58.566365 22910 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b4bf91c758b84f579814f360b1560d8c: Generated new TSK 0
I20260924 04:12:58.567369 22910 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 04:12:58.604843 22870 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:12:58.601614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35470 (local address 127.0.98.60:42761)
0924 04:12:58.601719 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:58.601724 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:58.603266 (+ 1542us) server_negotiation.cc:407] Connection header received
0924 04:12:58.603738 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:58.603742 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:58.603793 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:58.603849 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:12:58.604708 (+ 859us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35470: BlockingRecv error: recv got EOF from 127.0.0.1:35470 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260924 04:12:58.605582 22717 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:12:58.601140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34420 (local address 127.0.98.62:33555)
0924 04:12:58.601265 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:58.601270 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:58.603081 (+ 1811us) server_negotiation.cc:407] Connection header received
0924 04:12:58.603994 (+ 913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:58.603997 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:58.604044 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:58.604093 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:12:58.605465 (+ 1372us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34420: BlockingRecv error: recv got EOF from 127.0.0.1:34420 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260924 04:12:58.606616 22795 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:12:58.602013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48920 (local address 127.0.98.61:33925)
0924 04:12:58.602123 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:58.602127 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:58.603567 (+ 1440us) server_negotiation.cc:407] Connection header received
0924 04:12:58.605908 (+ 2341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:58.605911 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:58.605960 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:58.606022 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:12:58.606507 (+ 485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48920: BlockingRecv error: recv got EOF from 127.0.0.1:48920 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260924 04:12:58.614733 22795 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:12:58.613161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48926 (local address 127.0.98.61:33925)
0924 04:12:58.613268 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:58.613272 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:58.613729 (+ 457us) server_negotiation.cc:407] Connection header received
0924 04:12:58.613954 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:58.613957 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:58.614050 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:58.614103 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:12:58.614619 (+ 516us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48926: BlockingRecv error: recv got EOF from 127.0.0.1:48926 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260924 04:12:58.619872 22795 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:12:58.618409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48930 (local address 127.0.98.61:33925)
0924 04:12:58.618507 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:58.618512 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:58.618997 (+ 485us) server_negotiation.cc:407] Connection header received
0924 04:12:58.619180 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:58.619183 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:58.619232 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:58.619282 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:12:58.619769 (+ 487us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48930: BlockingRecv error: recv got EOF from 127.0.0.1:48930 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260924 04:12:58.625386 22795 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:12:58.623908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48932 (local address 127.0.98.61:33925)
0924 04:12:58.624017 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:58.624021 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:58.624512 (+ 491us) server_negotiation.cc:407] Connection header received
0924 04:12:58.624708 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:58.624711 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:58.624759 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:58.624810 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:12:58.625285 (+ 475us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48932: BlockingRecv error: recv got EOF from 127.0.0.1:48932 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260924 04:12:58.632936 22795 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:12:58.631425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48946 (local address 127.0.98.61:33925)
0924 04:12:58.631560 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:58.631564 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:58.632038 (+ 474us) server_negotiation.cc:407] Connection header received
0924 04:12:58.632223 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:58.632226 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:58.632273 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:58.632322 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:12:58.632832 (+ 510us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48946: BlockingRecv error: recv got EOF from 127.0.0.1:48946 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260924 04:12:58.644680 22795 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:12:58.643087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48950 (local address 127.0.98.61:33925)
0924 04:12:58.643191 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:58.643195 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:58.643729 (+ 534us) server_negotiation.cc:407] Connection header received
0924 04:12:58.643923 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:58.643927 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:58.643991 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:58.644043 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:12:58.644572 (+ 529us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48950: BlockingRecv error: recv got EOF from 127.0.0.1:48950 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260924 04:12:58.664978 22795 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:12:58.663092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48966 (local address 127.0.98.61:33925)
0924 04:12:58.663215 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:58.663219 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:58.663894 (+ 675us) server_negotiation.cc:407] Connection header received
0924 04:12:58.664125 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:58.664129 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:58.664188 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:58.664243 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:12:58.664846 (+ 603us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48966: BlockingRecv error: recv got EOF from 127.0.0.1:48966 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260924 04:12:58.701411 22795 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:12:58.699622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48982 (local address 127.0.98.61:33925)
0924 04:12:58.699747 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:58.699752 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:58.700355 (+ 603us) server_negotiation.cc:407] Connection header received
0924 04:12:58.700558 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:58.700562 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:58.700617 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:58.700669 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:12:58.701254 (+ 585us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48982: BlockingRecv error: recv got EOF from 127.0.0.1:48982 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260924 04:12:58.769127 22795 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:12:58.767930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48994 (local address 127.0.98.61:33925)
0924 04:12:58.768069 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:58.768073 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:58.768190 (+ 117us) server_negotiation.cc:407] Connection header received
0924 04:12:58.768353 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:58.768356 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:58.768405 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:58.768495 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:12:58.768972 (+ 477us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48994: BlockingRecv error: recv got EOF from 127.0.0.1:48994 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260924 04:12:58.900750 22795 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:12:58.899391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49006 (local address 127.0.98.61:33925)
0924 04:12:58.899562 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:58.899567 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:58.899717 (+ 150us) server_negotiation.cc:407] Connection header received
0924 04:12:58.899897 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:58.899900 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:58.900000 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:58.900100 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:12:58.900573 (+ 473us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49006: BlockingRecv error: recv got EOF from 127.0.0.1:49006 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260924 04:12:59.160048 22795 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:12:59.158942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49014 (local address 127.0.98.61:33925)
0924 04:12:59.159107 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:59.159111 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:59.159199 (+ 88us) server_negotiation.cc:407] Connection header received
0924 04:12:59.159372 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:59.159375 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:59.159429 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:59.159515 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:12:59.159943 (+ 428us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49014: BlockingRecv error: recv got EOF from 127.0.0.1:49014 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 04:12:59.481487 22893 catalog_manager.cc:1259] Loaded cluster ID: b52af584698c40b593370c331815e1d9
I20260924 04:12:59.481575 22893 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 523616d387004f04a984e8cb42ec8c78: loading cluster ID for follower catalog manager: success
I20260924 04:12:59.483299 22893 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 523616d387004f04a984e8cb42ec8c78: acquiring CA information for follower catalog manager: success
I20260924 04:12:59.484521 22893 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 523616d387004f04a984e8cb42ec8c78: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 04:12:59.520790 22928 catalog_manager.cc:1259] Loaded cluster ID: b52af584698c40b593370c331815e1d9
I20260924 04:12:59.520972 22928 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f96150155eec4ab9bd4678293cb33541: loading cluster ID for follower catalog manager: success
I20260924 04:12:59.522684 22928 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f96150155eec4ab9bd4678293cb33541: acquiring CA information for follower catalog manager: success
I20260924 04:12:59.523834 22928 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f96150155eec4ab9bd4678293cb33541: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 04:12:59.678552 22999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:59.674202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49606 (local address 127.0.98.61:33925)
0924 04:12:59.674529 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:59.674533 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:59.674549 (+ 16us) server_negotiation.cc:407] Connection header received
0924 04:12:59.674601 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:59.674604 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:59.674727 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:59.674842 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:12:59.675101 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:59.675840 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:59.677028 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:59.677304 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:59.677414 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:59.677684 (+ 270us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:12:59.678126 (+ 442us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:12:59.678223 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:59.678289 (+ 66us) server_negotiation.cc:299] Negotiation successful
0924 04:12:59.678339 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":72,"threads_started":1}
I20260924 04:12:59.684828 23005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:59.681608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40646 (local address 127.0.98.60:42761)
0924 04:12:59.681837 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:59.681841 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:59.681852 (+ 11us) server_negotiation.cc:407] Connection header received
0924 04:12:59.681927 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:59.681929 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:59.682039 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:59.682138 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:12:59.682312 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:59.682803 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:59.683579 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:59.683794 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:59.683864 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:59.684115 (+ 251us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:12:59.684504 (+ 389us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:12:59.684603 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:59.684657 (+ 54us) server_negotiation.cc:299] Negotiation successful
0924 04:12:59.684720 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":70,"threads_started":1}
I20260924 04:12:59.691880 22999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:59.687679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49618 (local address 127.0.98.61:33925)
0924 04:12:59.687794 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:59.687799 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:59.688340 (+ 541us) server_negotiation.cc:407] Connection header received
0924 04:12:59.688545 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:59.688548 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:59.688610 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:59.688674 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:12:59.688983 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:59.689725 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:59.690640 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:59.690962 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:59.691070 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:59.691381 (+ 311us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:12:59.691489 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:12:59.691563 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:59.691707 (+ 144us) server_negotiation.cc:299] Negotiation successful
0924 04:12:59.691760 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 04:12:59.691877 23005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:59.687893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40648 (local address 127.0.98.60:42761)
0924 04:12:59.687984 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:59.687988 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:59.688327 (+ 339us) server_negotiation.cc:407] Connection header received
0924 04:12:59.688458 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:59.688462 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:59.688539 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:59.688606 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:12:59.688956 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:59.689437 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:59.690685 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:59.691001 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:59.691086 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:59.691482 (+ 396us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:12:59.691559 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:12:59.691625 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:59.691712 (+ 87us) server_negotiation.cc:299] Negotiation successful
0924 04:12:59.691757 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 04:12:59.692373 23011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:12:59.687683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49092 (local address 127.0.98.62:33555)
0924 04:12:59.688075 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0924 04:12:59.688092 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 04:12:59.688108 (+ 16us) server_negotiation.cc:407] Connection header received
0924 04:12:59.688289 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:12:59.688296 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:12:59.688469 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:12:59.688578 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:12:59.688820 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:59.689655 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:59.690753 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:12:59.690997 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:12:59.691114 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:12:59.691517 (+ 403us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:12:59.691865 (+ 348us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:12:59.692117 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0924 04:12:59.692151 (+ 34us) server_negotiation.cc:299] Negotiation successful
0924 04:12:59.692208 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":93,"threads_started":1}
I20260924 04:12:59.701722 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 22646
I20260924 04:12:59.708112 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 22721
I20260924 04:12:59.713956 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 22796
2026-09-24T04:12:59Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3076 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222898807720-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:12:59 dist-test-slave-00ld krb5kdc[23016](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:12:59 dist-test-slave-00ld krb5kdc[23016](info): set up 2 sockets
Sep 24 04:12:59 dist-test-slave-00ld krb5kdc[23016](info): commencing operation
krb5kdc: starting...
W20260924 04:13:00.753288 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222898807720-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222898807720-392-0/krb5kdc/test-user.keytab.
Sep 24 04:13:00 dist-test-slave-00ld krb5kdc[23016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223180, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:13:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:13:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:13:00.880378 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222898807720-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222898807720-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222898807720-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33101
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33101,127.0.98.61:36631,127.0.98.60:40449
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:13:00.992312 23029 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 04:13:00.992616 23029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:13:00.992640 23029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:13:00.992728 23029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:13:00.996835 23029 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:13:00.996896 23029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:13:00.996915 23029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:13:00.996932 23029 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:13:00.996948 23029 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:13:01.002271 23029 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222898807720-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222898807720-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33101,127.0.98.61:36631,127.0.98.60:40449
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222898807720-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23029
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222898807720-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:13:01.003283 23029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:13:01.004282 23029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:13:01.013275 23035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:13:01.013255 23037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:13:01.013405 23034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:13:01.013669 23029 server_base.cc:1034] running on GCE node
I20260924 04:13:01.014189 23029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:13:01.015298 23029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:13:01.016485 23029 hybrid_clock.cc:648] HybridClock initialized: now 1790223181016471 us; error 29 us; skew 500 ppm
Sep 24 04:13:01 dist-test-slave-00ld krb5kdc[23016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223181, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:13:01.019186 23029 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:13:01.019973 23029 webserver.cc:460] Webserver started at http://127.0.98.62:32835/ using document root <none> and password file <none>
I20260924 04:13:01.020460 23029 fs_manager.cc:359] Metadata directory not provided
I20260924 04:13:01.020517 23029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:13:01.020735 23029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:13:01.022671 23029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222898807720-392-0/minicluster-data/master-0/data/instance:
uuid: "d1e2b25261fe4f129a81c81041c75a84"
format_stamp: "Formatted at 2026-09-24 04:13:01 on dist-test-slave-00ld"
server_key: "faac1a51b6e734bec4b3990a49da1beb"
server_key_iv: "f901288178dedee1953498296d8d8008"
server_key_version: "encryptionkey@0"
I20260924 04:13:01.023168 23029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222898807720-392-0/minicluster-data/master-0/wal/instance:
uuid: "d1e2b25261fe4f129a81c81041c75a84"
format_stamp: "Formatted at 2026-09-24 04:13:01 on dist-test-slave-00ld"
server_key: "faac1a51b6e734bec4b3990a49da1beb"
server_key_iv: "f901288178dedee1953498296d8d8008"
server_key_version: "encryptionkey@0"
I20260924 04:13:01.026903 23029 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260924 04:13:01.029241 23044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:13:01.030424 23029 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:13:01.030550 23029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222898807720-392-0/minicluster-data/master-0/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222898807720-392-0/minicluster-data/master-0/wal
uuid: "d1e2b25261fe4f129a81c81041c75a84"
format_stamp: "Formatted at 2026-09-24 04:13:01 on dist-test-slave-00ld"
server_key: "faac1a51b6e734bec4b3990a49da1beb"
server_key_iv: "f901288178dedee1953498296d8d8008"
server_key_version: "encryptionkey@0"
I20260924 04:13:01.030659 23029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222898807720-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222898807720-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222898807720-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:13:01.055681 23029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:13:01.060638 23029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:13:01.060886 23029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:13:01.070801 23029 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33101
I20260924 04:13:01.071151 23096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33101 every 8 connection(s)
I20260924 04:13:01.072188 23029 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222898807720-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:13:01.075763 23097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:13:01.077186 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 23029
I20260924 04:13:01.077354 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222898807720-392-0/minicluster-data/master-0/wal/instance
I20260924 04:13:01.077649 392 external_mini_cluster.cc:1428] Setting key d086307b9ccd1e94ee99b32063f031c1
I20260924 04:13:01.081188 23097 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33101 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 24 04:13:01 dist-test-slave-00ld krb5kdc[23016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223181, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:13:01.101382 23100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:01.084027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35494 (local address 127.0.98.62:33101)
0924 04:13:01.084532 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:01.084551 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:01.084952 (+ 401us) server_negotiation.cc:407] Connection header received
0924 04:13:01.086366 (+ 1414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:01.086370 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:01.086568 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:01.086765 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:13:01.087818 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:01.088999 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:01.090641 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:01.091010 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:01.091187 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:01.095496 (+ 4309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:13:01.095505 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:13:01.095525 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:13:01.095560 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:13:01.098013 (+ 2453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:01.098509 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:01.098518 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:01.098520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:01.098675 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:01.099045 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:01.099050 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:01.099054 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:01.099634 (+ 580us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:13:01.099790 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0924 04:13:01.100313 (+ 523us) server_negotiation.cc:299] Negotiation successful
0924 04:13:01.100614 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":57,"server-negotiator.queue_time_us":331,"thread_start_us":137,"threads_started":1}
I20260924 04:13:01.101411 23101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:01.084027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33101 (local address 127.0.0.1:35494)
0924 04:13:01.084752 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0924 04:13:01.084802 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 04:13:01.086019 (+ 1217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:13:01.086759 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:13:01.086764 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:13:01.087234 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:13:01.087641 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:01.087679 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:01.089122 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:01.089125 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:01.090505 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:01.090530 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:01.091174 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:01.091177 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:01.091282 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:01.093029 (+ 1747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:13:01.093056 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:13:01.095279 (+ 2223us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:13:01.098130 (+ 2851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:13:01.098142 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:13:01.098155 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:13:01.098377 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:13:01.098838 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:13:01.098840 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:13:01.098842 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:13:01.098945 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:13:01.099756 (+ 811us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:13:01.099795 (+ 39us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:13:01.100005 (+ 210us) client_negotiation.cc:769] Sending connection context
0924 04:13:01.100202 (+ 197us) client_negotiation.cc:240] Negotiation successful
0924 04:13:01.100465 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"spinlock_wait_cycles":78080,"thread_start_us":137,"threads_started":1}
I20260924 04:13:01.103917 23097 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36631 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:13:01.104652 23101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:01.104413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36631 (local address 127.0.0.1:47734)
0924 04:13:01.104523 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0924 04:13:01.104598 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260924 04:13:01.105767 23045 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36631: connect: Connection refused (error 111)
I20260924 04:13:01.106554 23101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:01.106373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36631 (local address 127.0.0.1:47738)
0924 04:13:01.106467 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0924 04:13:01.106508 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260924 04:13:01.106727 23097 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:36631: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36631: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222898807720-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222898807720-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260924 04:13:01.107645 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222898807720-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222898807720-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222898807720-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222898807720-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:36631
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222898807720-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33101,127.0.98.61:36631,127.0.98.60:40449
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:13:01.156008 23097 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36631 } attempt: 1
I20260924 04:13:01.156973 23101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:01.156656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36631 (local address 127.0.0.1:47744)
0924 04:13:01.156832 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 04:13:01.156895 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260924 04:13:01.157775 23101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:01.157551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36631 (local address 127.0.0.1:47748)
0924 04:13:01.157663 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 04:13:01.157722 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260924 04:13:01.157905 23097 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:36631: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36631: connect: Connection refused (error 111)
W20260924 04:13:01.218592 23104 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 04:13:01.218940 23104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:13:01.219010 23104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:13:01.219089 23104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:13:01.223163 23104 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:13:01.223256 23104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:13:01.223295 23104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:13:01.223354 23104 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:13:01.223395 23104 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:13:01.226064 23097 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36631 } attempt: 2
I20260924 04:13:01.226836 23101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:01.226551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36631 (local address 127.0.0.1:47752)
0924 04:13:01.226698 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 04:13:01.226764 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260924 04:13:01.227557 23101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:01.227335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36631 (local address 127.0.0.1:47756)
0924 04:13:01.227444 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0924 04:13:01.227494 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260924 04:13:01.227670 23097 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:36631: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36631: connect: Connection refused (error 111)
I20260924 04:13:01.229539 23104 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222898807720-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222898807720-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33101,127.0.98.61:36631,127.0.98.60:40449
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222898807720-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:36631
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222898807720-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23104
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222898807720-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:13:01.230684 23104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:13:01.231738 23104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:13:01.240247 23112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:13:01.240247 23110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:13:01.240266 23109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:13:01.240625 23104 server_base.cc:1034] running on GCE node
I20260924 04:13:01.241165 23104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:13:01.242238 23104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:13:01.243444 23104 hybrid_clock.cc:648] HybridClock initialized: now 1790223181243425 us; error 27 us; skew 500 ppm
Sep 24 04:13:01 dist-test-slave-00ld krb5kdc[23016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223181, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:13:01.246335 23104 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260924 04:13:01.247218 23104 webserver.cc:460] Webserver started at http://127.0.98.61:39673/ using document root <none> and password file <none>
I20260924 04:13:01.247743 23104 fs_manager.cc:359] Metadata directory not provided
I20260924 04:13:01.247802 23104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:13:01.248001 23104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:13:01.249986 23104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222898807720-392-0/minicluster-data/master-1/data/instance:
uuid: "fd728e053cd4403cb43f85661c627ada"
format_stamp: "Formatted at 2026-09-24 04:13:01 on dist-test-slave-00ld"
server_key: "0078430a277cf850f77d97bdd4b9282a"
server_key_iv: "f018fe1ac4558fbf85f0b7befb5ab5a5"
server_key_version: "encryptionkey@0"
I20260924 04:13:01.250535 23104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222898807720-392-0/minicluster-data/master-1/wal/instance:
uuid: "fd728e053cd4403cb43f85661c627ada"
format_stamp: "Formatted at 2026-09-24 04:13:01 on dist-test-slave-00ld"
server_key: "0078430a277cf850f77d97bdd4b9282a"
server_key_iv: "f018fe1ac4558fbf85f0b7befb5ab5a5"
server_key_version: "encryptionkey@0"
I20260924 04:13:01.254101 23104 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:13:01.256529 23119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:13:01.257766 23104 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:13:01.257916 23104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222898807720-392-0/minicluster-data/master-1/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222898807720-392-0/minicluster-data/master-1/wal
uuid: "fd728e053cd4403cb43f85661c627ada"
format_stamp: "Formatted at 2026-09-24 04:13:01 on dist-test-slave-00ld"
server_key: "0078430a277cf850f77d97bdd4b9282a"
server_key_iv: "f018fe1ac4558fbf85f0b7befb5ab5a5"
server_key_version: "encryptionkey@0"
I20260924 04:13:01.258033 23104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222898807720-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222898807720-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222898807720-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:13:01.270489 23104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:13:01.273442 23104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:13:01.273643 23104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:13:01.280288 23104 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:36631
I20260924 04:13:01.280555 23171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:36631 every 8 connection(s)
I20260924 04:13:01.281255 23104 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222898807720-392-0/minicluster-data/master-1/data/info.pb
I20260924 04:13:01.283586 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 23104
I20260924 04:13:01.283740 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222898807720-392-0/minicluster-data/master-1/wal/instance
I20260924 04:13:01.284035 392 external_mini_cluster.cc:1428] Setting key 2a5269200d56d27add57bd97fe930200
I20260924 04:13:01.284077 23172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:13:01.287838 23172 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33101 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 24 04:13:01 dist-test-slave-00ld krb5kdc[23016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223181, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:13:01.301805 23100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:01.289751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35498 (local address 127.0.98.62:33101)
0924 04:13:01.289882 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:01.289885 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:01.290869 (+ 984us) server_negotiation.cc:407] Connection header received
0924 04:13:01.291828 (+ 959us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:01.291832 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:01.291895 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:01.292001 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:13:01.293556 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:01.294091 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:01.295450 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:01.295696 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:01.295808 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:01.298251 (+ 2443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:13:01.298254 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:13:01.298256 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:13:01.298280 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:13:01.299722 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:01.300224 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:01.300228 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:01.300230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:01.300310 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:01.300643 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:01.300646 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:01.300648 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:01.300876 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:13:01.300980 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 04:13:01.301491 (+ 511us) server_negotiation.cc:299] Negotiation successful
0924 04:13:01.301627 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 04:13:01.302647 23175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:01.290245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33101 (local address 127.0.0.1:35498)
0924 04:13:01.290720 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0924 04:13:01.290760 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 04:13:01.291622 (+ 862us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:13:01.292166 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:13:01.292173 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:13:01.292501 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:13:01.293392 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:01.293410 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:01.294222 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:01.294228 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:01.295324 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:01.295334 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:01.295847 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:01.295851 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:01.296028 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:01.296624 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:13:01.296640 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:13:01.298071 (+ 1431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:13:01.299844 (+ 1773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:13:01.299850 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:13:01.299864 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:13:01.300121 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:13:01.300401 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:13:01.300405 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:13:01.300407 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:13:01.300544 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:13:01.300980 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:13:01.300987 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:13:01.301247 (+ 260us) client_negotiation.cc:769] Sending connection context
0924 04:13:01.301437 (+ 190us) client_negotiation.cc:240] Negotiation successful
0924 04:13:01.301750 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":134,"threads_started":1}
I20260924 04:13:01.304136 23172 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36631 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 04:13:01 dist-test-slave-00ld krb5kdc[23016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223181, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222898807720-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222898807720-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260924 04:13:01.311184 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1CVu5m/build/debug/bin/kudu
/tmp/dist-test-task1CVu5m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222898807720-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222898807720-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222898807720-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222898807720-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:40449
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222898807720-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33101,127.0.98.61:36631,127.0.98.60:40449
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222898807720-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222898807720-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222898807720-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:13:01.313164 23175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:01.304646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36631 (local address 127.0.0.1:47764)
0924 04:13:01.304787 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 04:13:01.304799 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:13:01.304864 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:13:01.305496 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:13:01.305498 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:13:01.305653 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:13:01.305736 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:01.305741 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:01.306684 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:01.306686 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:01.307243 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:01.307251 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:01.307852 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:01.307863 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:01.307933 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:01.308340 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:13:01.308351 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:13:01.309612 (+ 1261us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:13:01.311582 (+ 1970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:13:01.311589 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:13:01.311594 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:13:01.311857 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:13:01.312130 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:13:01.312136 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:13:01.312141 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:13:01.312242 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:13:01.312778 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:13:01.312781 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:13:01.312866 (+ 85us) client_negotiation.cc:769] Sending connection context
0924 04:13:01.312950 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 04:13:01.313050 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260924 04:13:01.313596 23177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:01.304786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47764 (local address 127.0.98.61:36631)
0924 04:13:01.305083 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:01.305094 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:01.305143 (+ 49us) server_negotiation.cc:407] Connection header received
0924 04:13:01.305206 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:01.305213 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:01.305380 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:01.305481 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:13:01.305815 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:01.306589 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:01.307378 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:01.307716 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:01.307796 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:01.309727 (+ 1931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:13:01.309735 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:13:01.309745 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:13:01.309765 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:13:01.311479 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:01.311967 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:01.311971 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:01.311972 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:01.312042 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:01.312330 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:01.312332 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:01.312333 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:01.312649 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:13:01.312814 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0924 04:13:01.313065 (+ 251us) server_negotiation.cc:299] Negotiation successful
0924 04:13:01.313242 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":136,"threads_started":1}
I20260924 04:13:01.315292 23172 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40449 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:13:01.315989 23175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:01.315759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40449 (local address 127.0.0.1:41992)
0924 04:13:01.315865 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0924 04:13:01.315940 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260924 04:13:01.316169 23122 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40449: connect: Connection refused (error 111)
I20260924 04:13:01.317039 23175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:01.316850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40449 (local address 127.0.0.1:42000)
0924 04:13:01.316944 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0924 04:13:01.316989 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260924 04:13:01.317189 23172 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40449: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40449: connect: Connection refused (error 111)
I20260924 04:13:01.318809 23097 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36631 } attempt: 3
Sep 24 04:13:01 dist-test-slave-00ld krb5kdc[23016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223181, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 04:13:01.326988 23101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:01.319230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36631 (local address 127.0.0.1:47768)
0924 04:13:01.319349 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0924 04:13:01.319363 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:13:01.319453 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:13:01.319694 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:13:01.319697 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:13:01.319861 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:13:01.319958 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:01.319963 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:01.320743 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:01.320746 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:01.321666 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:01.321674 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:01.322145 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:01.322148 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:01.322251 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:01.322727 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:13:01.322747 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:13:01.323898 (+ 1151us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:13:01.325562 (+ 1664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:13:01.325567 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:13:01.325570 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:13:01.325762 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:13:01.326043 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:13:01.326046 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:13:01.326048 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:13:01.326135 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:13:01.326483 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:13:01.326486 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:13:01.326586 (+ 100us) client_negotiation.cc:769] Sending connection context
0924 04:13:01.326693 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 04:13:01.326831 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260924 04:13:01.326988 23177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:01.319261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47768 (local address 127.0.98.61:36631)
0924 04:13:01.319349 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:01.319354 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:01.319408 (+ 54us) server_negotiation.cc:407] Connection header received
0924 04:13:01.319548 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:01.319552 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:01.319590 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:01.319679 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:13:01.320070 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:01.320580 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:01.321784 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:01.322036 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:01.322141 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:01.324023 (+ 1882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:13:01.324026 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:13:01.324029 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:13:01.324051 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:13:01.325454 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:01.325858 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:01.325863 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:01.325865 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:01.325938 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:01.326224 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:01.326228 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:01.326230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:01.326391 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:13:01.326492 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 04:13:01.326741 (+ 249us) server_negotiation.cc:299] Negotiation successful
0924 04:13:01.326842 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 04:13:01.328022 23097 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40449 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:13:01.328850 23101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:01.328514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40449 (local address 127.0.0.1:42014)
0924 04:13:01.328657 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 04:13:01.328775 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260924 04:13:01.329538 23101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:01.329290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40449 (local address 127.0.0.1:42028)
0924 04:13:01.329430 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 04:13:01.329481 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260924 04:13:01.329660 23097 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40449: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40449: connect: Connection refused (error 111)
I20260924 04:13:01.360841 23097 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40449 } attempt: 1
I20260924 04:13:01.361811 23101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:01.361517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40449 (local address 127.0.0.1:42044)
0924 04:13:01.361658 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 04:13:01.361726 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260924 04:13:01.362692 23101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:01.362477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40449 (local address 127.0.0.1:42048)
0924 04:13:01.362572 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0924 04:13:01.362629 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260924 04:13:01.362831 23097 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40449: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40449: connect: Connection refused (error 111)
I20260924 04:13:01.366437 23172 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40449 } attempt: 1
I20260924 04:13:01.367209 23175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:01.366926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40449 (local address 127.0.0.1:42064)
0924 04:13:01.367077 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 04:13:01.367143 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260924 04:13:01.367959 23175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:01.367724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40449 (local address 127.0.0.1:42074)
0924 04:13:01.367847 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 04:13:01.367903 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260924 04:13:01.368093 23172 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40449: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40449: connect: Connection refused (error 111)
W20260924 04:13:01.429322 23178 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 04:13:01.429689 23178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:13:01.429754 23178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:13:01.429834 23178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:13:01.435377 23178 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:13:01.435510 23178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:13:01.435554 23178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:13:01.435592 23178 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:13:01.435628 23178 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:13:01.438344 23097 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40449 } attempt: 2
I20260924 04:13:01.438647 23172 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40449 } attempt: 2
I20260924 04:13:01.439220 23101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:01.438884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40449 (local address 127.0.0.1:42086)
0924 04:13:01.439047 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 04:13:01.439129 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260924 04:13:01.439303 23175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:01.439002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40449 (local address 127.0.0.1:42096)
0924 04:13:01.439175 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 04:13:01.439225 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260924 04:13:01.439881 23101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:01.439657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40449 (local address 127.0.0.1:42098)
0924 04:13:01.439773 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0924 04:13:01.439819 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260924 04:13:01.440034 23097 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40449: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40449: connect: Connection refused (error 111)
I20260924 04:13:01.440332 23175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:01.440129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40449 (local address 127.0.0.1:42110)
0924 04:13:01.440220 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0924 04:13:01.440269 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260924 04:13:01.440477 23172 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40449: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40449: connect: Connection refused (error 111)
I20260924 04:13:01.443246 23178 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222898807720-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222898807720-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33101,127.0.98.61:36631,127.0.98.60:40449
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222898807720-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:40449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222898807720-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23178
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222898807720-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 03:13:04 UTC on ccycloud.quasar-pgpacm.root.comops.site
build id 4669246
I20260924 04:13:01.444417 23178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:13:01.445537 23178 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:13:01.454378 23187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:13:01.454412 23185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:13:01.454412 23184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:13:01.454665 23178 server_base.cc:1034] running on GCE node
I20260924 04:13:01.455279 23178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:13:01.456463 23178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:13:01.457686 23178 hybrid_clock.cc:648] HybridClock initialized: now 1790223181457655 us; error 44 us; skew 500 ppm
Sep 24 04:13:01 dist-test-slave-00ld krb5kdc[23016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223181, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:13:01.460644 23178 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260924 04:13:01.461761 23178 webserver.cc:460] Webserver started at http://127.0.98.60:45265/ using document root <none> and password file <none>
I20260924 04:13:01.462345 23178 fs_manager.cc:359] Metadata directory not provided
I20260924 04:13:01.462440 23178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:13:01.462677 23178 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:13:01.464800 23178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222898807720-392-0/minicluster-data/master-2/data/instance:
uuid: "14c03ef6d11244808c7af265f436dbf8"
format_stamp: "Formatted at 2026-09-24 04:13:01 on dist-test-slave-00ld"
server_key: "b5dd869a1c28dc32b5b691cd46df80dd"
server_key_iv: "fd3c1e33f6408dee0dbf81f1e9f1bbe8"
server_key_version: "encryptionkey@0"
I20260924 04:13:01.465355 23178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222898807720-392-0/minicluster-data/master-2/wal/instance:
uuid: "14c03ef6d11244808c7af265f436dbf8"
format_stamp: "Formatted at 2026-09-24 04:13:01 on dist-test-slave-00ld"
server_key: "b5dd869a1c28dc32b5b691cd46df80dd"
server_key_iv: "fd3c1e33f6408dee0dbf81f1e9f1bbe8"
server_key_version: "encryptionkey@0"
I20260924 04:13:01.469060 23178 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 04:13:01.471450 23194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:13:01.472862 23178 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 04:13:01.473008 23178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222898807720-392-0/minicluster-data/master-2/data,/tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222898807720-392-0/minicluster-data/master-2/wal
uuid: "14c03ef6d11244808c7af265f436dbf8"
format_stamp: "Formatted at 2026-09-24 04:13:01 on dist-test-slave-00ld"
server_key: "b5dd869a1c28dc32b5b691cd46df80dd"
server_key_iv: "fd3c1e33f6408dee0dbf81f1e9f1bbe8"
server_key_version: "encryptionkey@0"
I20260924 04:13:01.473124 23178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222898807720-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222898807720-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222898807720-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:13:01.504462 23178 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:13:01.507561 23178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:13:01.507774 23178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:13:01.514870 23178 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:40449
I20260924 04:13:01.515251 23246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:40449 every 8 connection(s)
I20260924 04:13:01.515866 23178 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222898807720-392-0/minicluster-data/master-2/data/info.pb
I20260924 04:13:01.518653 23247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:13:01.519675 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1CVu5m/build/debug/bin/kudu as pid 23178
I20260924 04:13:01.519817 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1CVu5m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790222898807720-392-0/minicluster-data/master-2/wal/instance
I20260924 04:13:01.520118 392 external_mini_cluster.cc:1428] Setting key 9ff7acb03602f6189f9cbbe76cf5aaf7
I20260924 04:13:01.523113 23247 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33101 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 04:13:01 dist-test-slave-00ld krb5kdc[23016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223180, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 24 04:13:01 dist-test-slave-00ld krb5kdc[23016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223181, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:13:01.531620 23172 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40449 } attempt: 3
I20260924 04:13:01.534989 23100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:01.525086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35502 (local address 127.0.98.62:33101)
0924 04:13:01.525236 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:01.525241 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:01.525542 (+ 301us) server_negotiation.cc:407] Connection header received
0924 04:13:01.525662 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:01.525665 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:01.525718 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:01.525795 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:13:01.526471 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:01.527073 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:01.528327 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:01.528735 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:01.528863 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:01.531219 (+ 2356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:13:01.531223 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:13:01.531226 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:13:01.531254 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:13:01.532969 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:01.533510 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:01.533514 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:01.533516 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:01.533602 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:01.533888 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:01.533891 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:01.533893 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:01.534162 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:13:01.534276 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 04:13:01.534761 (+ 485us) server_negotiation.cc:299] Negotiation successful
0924 04:13:01.534863 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 04:13:01.535881 23250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:01.521503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42122 (local address 127.0.98.60:40449)
0924 04:13:01.521894 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:01.521915 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:01.521973 (+ 58us) server_negotiation.cc:407] Connection header received
0924 04:13:01.522795 (+ 822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:01.522818 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:01.523255 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:01.523622 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:13:01.524320 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:01.525870 (+ 1550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:01.526815 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:01.527508 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:01.527658 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:01.530392 (+ 2734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:13:01.530397 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:13:01.530407 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:13:01.530428 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:13:01.532392 (+ 1964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:01.532946 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:01.532955 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:01.532962 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:01.533103 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:01.533434 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:01.533437 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:01.533438 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:01.533915 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:13:01.534101 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0924 04:13:01.534512 (+ 411us) server_negotiation.cc:299] Negotiation successful
0924 04:13:01.534799 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":100,"threads_started":1}
I20260924 04:13:01.536163 23251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:01.525029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33101 (local address 127.0.0.1:35502)
0924 04:13:01.525403 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0924 04:13:01.525449 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 04:13:01.525573 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:13:01.525865 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:13:01.525870 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:13:01.526228 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:13:01.526344 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:01.526349 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:01.527290 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:01.527294 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:01.528201 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:01.528212 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:01.528897 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:01.528900 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:01.529029 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:01.529524 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:13:01.529540 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:13:01.531075 (+ 1535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:13:01.533068 (+ 1993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:13:01.533076 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:13:01.533081 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:13:01.533403 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:13:01.533696 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:13:01.533700 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:13:01.533702 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:13:01.533790 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:13:01.534330 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:13:01.534348 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:13:01.534525 (+ 177us) client_negotiation.cc:769] Sending connection context
0924 04:13:01.535152 (+ 627us) client_negotiation.cc:240] Negotiation successful
0924 04:13:01.535330 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"spinlock_wait_cycles":768,"thread_start_us":141,"threads_started":1}
Sep 24 04:13:01 dist-test-slave-00ld krb5kdc[23016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223181, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 04:13:01.537598 23247 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36631 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:13:01.539188 23097 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40449 } attempt: 3
I20260924 04:13:01.541636 23175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:01.532106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40449 (local address 127.0.0.1:42136)
0924 04:13:01.532383 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0924 04:13:01.532398 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:13:01.532482 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:13:01.532988 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:13:01.532991 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:13:01.533194 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:13:01.533301 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:01.533308 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:01.534356 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:01.534359 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:01.535108 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:01.535121 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:01.535707 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:01.535711 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:01.535792 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:01.536328 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:13:01.536342 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:13:01.537831 (+ 1489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:13:01.540165 (+ 2334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:13:01.540169 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:13:01.540171 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:13:01.540370 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:13:01.540658 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:13:01.540662 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:13:01.540664 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:13:01.540778 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:13:01.541142 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:13:01.541145 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:13:01.541241 (+ 96us) client_negotiation.cc:769] Sending connection context
0924 04:13:01.541347 (+ 106us) client_negotiation.cc:240] Negotiation successful
0924 04:13:01.541473 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196}
I20260924 04:13:01.541674 23252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:01.532390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42136 (local address 127.0.98.60:40449)
0924 04:13:01.532737 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:01.532750 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:01.532764 (+ 14us) server_negotiation.cc:407] Connection header received
0924 04:13:01.532798 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:01.532800 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:01.532841 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:01.533427 (+ 586us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:13:01.533519 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:01.534233 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:01.535254 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:01.535553 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:01.535652 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:01.537982 (+ 2330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:13:01.537985 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:13:01.537987 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:13:01.538006 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:13:01.540056 (+ 2050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:01.540469 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:01.540474 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:01.540476 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:01.540561 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:01.540877 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:01.540880 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:01.540882 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:01.541046 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:13:01.541150 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 04:13:01.541407 (+ 257us) server_negotiation.cc:299] Negotiation successful
0924 04:13:01.541511 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":109,"threads_started":1}
Sep 24 04:13:01 dist-test-slave-00ld krb5kdc[23016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223180, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 24 04:13:01 dist-test-slave-00ld krb5kdc[23016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223181, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 04:13:01.545342 23172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd728e053cd4403cb43f85661c627ada: Bootstrap starting.
Sep 24 04:13:01 dist-test-slave-00ld krb5kdc[23016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223181, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 04:13:01.548487 23172 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd728e053cd4403cb43f85661c627ada: Neither blocks nor log segments found. Creating new log.
I20260924 04:13:01.548746 23100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:01.538516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35512 (local address 127.0.98.62:33101)
0924 04:13:01.538627 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:01.538631 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:01.538643 (+ 12us) server_negotiation.cc:407] Connection header received
0924 04:13:01.538744 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:01.538750 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:01.538797 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:01.538877 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:13:01.539760 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:01.540432 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:01.541534 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:01.541856 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:01.542020 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:01.545090 (+ 3070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:13:01.545094 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:13:01.545096 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:13:01.545123 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:13:01.546759 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:01.547245 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:01.547249 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:01.547250 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:01.547331 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:01.547642 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:01.547646 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:01.547647 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:01.547847 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:13:01.548033 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0924 04:13:01.548403 (+ 370us) server_negotiation.cc:299] Negotiation successful
0924 04:13:01.548525 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":48}
I20260924 04:13:01.549968 23172 log.cc:824] T 00000000000000000000000000000000 P fd728e053cd4403cb43f85661c627ada: Log is configured to *not* fsync() on all Append() calls
I20260924 04:13:01.550314 23177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:01.538187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47774 (local address 127.0.98.61:36631)
0924 04:13:01.538329 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:01.538334 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:01.538567 (+ 233us) server_negotiation.cc:407] Connection header received
0924 04:13:01.538718 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:01.538721 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:01.538766 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:01.538849 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:13:01.539351 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:01.540085 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:01.542123 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:01.542485 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:01.542571 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:01.545884 (+ 3313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:13:01.545890 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:13:01.545894 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:13:01.545920 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:13:01.548005 (+ 2085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:01.548575 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:01.548581 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:01.548584 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:01.548701 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:01.549052 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:01.549056 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:01.549060 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:01.549319 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:13:01.549396 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 04:13:01.550030 (+ 634us) server_negotiation.cc:299] Negotiation successful
0924 04:13:01.550147 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 04:13:01.550880 23251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:01.538085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36631 (local address 127.0.0.1:47774)
0924 04:13:01.538246 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 04:13:01.538513 (+ 267us) client_negotiation.cc:174] Beginning negotiation
0924 04:13:01.538627 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:13:01.538861 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:13:01.538864 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:13:01.539083 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:13:01.539213 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:01.539222 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:01.540543 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:01.540546 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:01.541378 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:01.541388 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:01.542720 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:01.542724 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:01.542864 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:01.544203 (+ 1339us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:13:01.544216 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:13:01.545704 (+ 1488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:13:01.548126 (+ 2422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:13:01.548130 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:13:01.548133 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:13:01.548445 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:13:01.548830 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:13:01.548834 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:13:01.548837 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:13:01.548937 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:13:01.549488 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:13:01.549494 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:13:01.549591 (+ 97us) client_negotiation.cc:769] Sending connection context
0924 04:13:01.550567 (+ 976us) client_negotiation.cc:240] Negotiation successful
0924 04:13:01.550702 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20260924 04:13:01.551599 23101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:01.539550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40449 (local address 127.0.0.1:42138)
0924 04:13:01.540223 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0924 04:13:01.540236 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:13:01.540295 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:13:01.541739 (+ 1444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:13:01.541742 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:13:01.541915 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:13:01.542165 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:01.542170 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:01.543126 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:01.543133 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:01.544023 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:01.544033 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:01.544631 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:01.544637 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:01.544768 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:01.545557 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:13:01.545573 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:13:01.547594 (+ 2021us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:13:01.550030 (+ 2436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:13:01.550033 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:13:01.550036 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:13:01.550270 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:13:01.550596 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:13:01.550599 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:13:01.550601 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:13:01.550689 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:13:01.551140 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:13:01.551143 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:13:01.551236 (+ 93us) client_negotiation.cc:769] Sending connection context
0924 04:13:01.551324 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 04:13:01.551420 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"mutex_wait_us":134}
I20260924 04:13:01.551617 23247 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40449 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:13:01.552321 23250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:01.540412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42138 (local address 127.0.98.60:40449)
0924 04:13:01.541553 (+ 1141us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:01.541557 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:01.541567 (+ 10us) server_negotiation.cc:407] Connection header received
0924 04:13:01.541595 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:01.541598 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:01.541633 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:01.542010 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:13:01.542280 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:01.542981 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:01.544146 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:01.544488 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:01.546915 (+ 2427us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:01.547748 (+ 833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:13:01.547752 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:13:01.547755 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:13:01.547782 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:13:01.549898 (+ 2116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:01.550397 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:01.550400 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:01.550401 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:01.550481 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:01.550797 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:01.550800 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:01.550802 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:01.551022 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:13:01.551750 (+ 728us) server_negotiation.cc:1035] Waiting for connection context
0924 04:13:01.551819 (+ 69us) server_negotiation.cc:299] Negotiation successful
0924 04:13:01.551938 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1076}
I20260924 04:13:01.553429 23172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd728e053cd4403cb43f85661c627ada: No bootstrap required, opened a new log
Sep 24 04:13:01 dist-test-slave-00ld krb5kdc[23016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223181, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 24 04:13:01 dist-test-slave-00ld krb5kdc[23016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790223180, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 04:13:01.557984 23172 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd728e053cd4403cb43f85661c627ada [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e2b25261fe4f129a81c81041c75a84" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33101 } } peers { permanent_uuid: "fd728e053cd4403cb43f85661c627ada" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36631 } } peers { permanent_uuid: "14c03ef6d11244808c7af265f436dbf8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40449 } }
I20260924 04:13:01.558252 23172 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd728e053cd4403cb43f85661c627ada [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd728e053cd4403cb43f85661c627ada, State: Initialized, Role: FOLLOWER
I20260924 04:13:01.558616 23172 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd728e053cd4403cb43f85661c627ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e2b25261fe4f129a81c81041c75a84" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33101 } } peers { permanent_uuid: "fd728e053cd4403cb43f85661c627ada" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36631 } } peers { permanent_uuid: "14c03ef6d11244808c7af265f436dbf8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40449 } }
I20260924 04:13:01.559136 23254 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd728e053cd4403cb43f85661c627ada [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e2b25261fe4f129a81c81041c75a84" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33101 } } peers { permanent_uuid: "fd728e053cd4403cb43f85661c627ada" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36631 } } peers { permanent_uuid: "14c03ef6d11244808c7af265f436dbf8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40449 } } }
I20260924 04:13:01.559363 23254 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd728e053cd4403cb43f85661c627ada [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:13:01.559561 23097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1e2b25261fe4f129a81c81041c75a84: Bootstrap starting.
I20260924 04:13:01.560022 23172 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd728e053cd4403cb43f85661c627ada [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:13:01.561152 23177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:01.550642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47780 (local address 127.0.98.61:36631)
0924 04:13:01.550751 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:01.550754 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:01.550765 (+ 11us) server_negotiation.cc:407] Connection header received
0924 04:13:01.550794 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:01.550797 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:01.550838 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:01.550925 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:13:01.551651 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:01.552369 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:01.553719 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:01.554036 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:01.554137 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:01.557755 (+ 3618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:13:01.557758 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:13:01.557761 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:13:01.557791 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:13:01.559411 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:01.559898 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:01.559902 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:01.559904 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:01.560003 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:01.560299 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:01.560303 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:01.560304 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:01.560486 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:13:01.560681 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0924 04:13:01.560886 (+ 205us) server_negotiation.cc:299] Negotiation successful
0924 04:13:01.561003 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 04:13:01.562685 23097 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d1e2b25261fe4f129a81c81041c75a84: Neither blocks nor log segments found. Creating new log.
I20260924 04:13:01.563651 23097 log.cc:824] T 00000000000000000000000000000000 P d1e2b25261fe4f129a81c81041c75a84: Log is configured to *not* fsync() on all Append() calls
I20260924 04:13:01.564813 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:13:01.566933 23251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:01.552040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40449 (local address 127.0.0.1:42152)
0924 04:13:01.552398 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0924 04:13:01.552416 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 04:13:01.552502 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:13:01.552813 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:13:01.552815 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:13:01.552956 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:13:01.553022 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:01.553027 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:01.553751 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:01.553754 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:01.554463 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:13:01.554474 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:13:01.555053 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:13:01.555056 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:13:01.555112 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:01.555512 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:13:01.555525 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:13:01.556750 (+ 1225us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:13:01.565459 (+ 8709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:13:01.565462 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:13:01.565464 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:13:01.565711 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:13:01.565988 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:13:01.565994 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:13:01.565995 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:13:01.566096 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:13:01.566507 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:13:01.566510 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:13:01.566577 (+ 67us) client_negotiation.cc:769] Sending connection context
0924 04:13:01.566650 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 04:13:01.566770 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287}
I20260924 04:13:01.563862 23104 master_runner.cc:186] Error getting master registration for 127.0.98.62:33101: OK, instance_id { permanent_uuid: "d1e2b25261fe4f129a81c81041c75a84" instance_seqno: 1790223181068599 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 04:13:01.567313 23104 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 04:13:01.566046 23071 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "d1e2b25261fe4f129a81c81041c75a84"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35512
I20260924 04:13:01.567551 23097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1e2b25261fe4f129a81c81041c75a84: No bootstrap required, opened a new log
I20260924 04:13:01.567646 23252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:01.552065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42152 (local address 127.0.98.60:40449)
0924 04:13:01.552162 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:01.552167 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:01.552567 (+ 400us) server_negotiation.cc:407] Connection header received
0924 04:13:01.552591 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:01.552593 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:01.552627 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:01.552727 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:13:01.553134 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:01.553621 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:01.554588 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:01.554940 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:01.555044 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:01.556880 (+ 1836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:13:01.556883 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:13:01.556889 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:13:01.556919 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:13:01.565335 (+ 8416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:01.565825 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:01.565829 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:01.565831 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:01.565913 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:01.566198 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:01.566202 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:01.566204 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:01.566411 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:13:01.567317 (+ 906us) server_negotiation.cc:1035] Waiting for connection context
0924 04:13:01.567391 (+ 74us) server_negotiation.cc:299] Negotiation successful
0924 04:13:01.567498 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
W20260924 04:13:01.568329 23265 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fd728e053cd4403cb43f85661c627ada: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:13:01.568401 23265 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:13:01.570567 23097 raft_consensus.cc:348] T 00000000000000000000000000000000 P d1e2b25261fe4f129a81c81041c75a84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e2b25261fe4f129a81c81041c75a84" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33101 } } peers { permanent_uuid: "fd728e053cd4403cb43f85661c627ada" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36631 } } peers { permanent_uuid: "14c03ef6d11244808c7af265f436dbf8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40449 } }
I20260924 04:13:01.570725 23247 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14c03ef6d11244808c7af265f436dbf8: Bootstrap starting.
I20260924 04:13:01.570827 23097 raft_consensus.cc:729] T 00000000000000000000000000000000 P d1e2b25261fe4f129a81c81041c75a84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1e2b25261fe4f129a81c81041c75a84, State: Initialized, Role: FOLLOWER
I20260924 04:13:01.571161 23097 consensus_queue.cc:260] T 00000000000000000000000000000000 P d1e2b25261fe4f129a81c81041c75a84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e2b25261fe4f129a81c81041c75a84" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33101 } } peers { permanent_uuid: "fd728e053cd4403cb43f85661c627ada" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36631 } } peers { permanent_uuid: "14c03ef6d11244808c7af265f436dbf8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40449 } }
I20260924 04:13:01.571671 23268 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1e2b25261fe4f129a81c81041c75a84 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e2b25261fe4f129a81c81041c75a84" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33101 } } peers { permanent_uuid: "fd728e053cd4403cb43f85661c627ada" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36631 } } peers { permanent_uuid: "14c03ef6d11244808c7af265f436dbf8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40449 } } }
I20260924 04:13:01.571898 23268 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1e2b25261fe4f129a81c81041c75a84 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:13:01.572468 23097 sys_catalog.cc:564] T 00000000000000000000000000000000 P d1e2b25261fe4f129a81c81041c75a84 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:13:01.572499 23071 raft_consensus.cc:482] T 00000000000000000000000000000000 P d1e2b25261fe4f129a81c81041c75a84 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 04:13:01.572902 23071 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d1e2b25261fe4f129a81c81041c75a84 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:13:01.573338 23247 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 14c03ef6d11244808c7af265f436dbf8: Neither blocks nor log segments found. Creating new log.
I20260924 04:13:01.574177 23247 log.cc:824] T 00000000000000000000000000000000 P 14c03ef6d11244808c7af265f436dbf8: Log is configured to *not* fsync() on all Append() calls
I20260924 04:13:01.574216 23071 raft_consensus.cc:504] T 00000000000000000000000000000000 P d1e2b25261fe4f129a81c81041c75a84 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e2b25261fe4f129a81c81041c75a84" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33101 } } peers { permanent_uuid: "fd728e053cd4403cb43f85661c627ada" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36631 } } peers { permanent_uuid: "14c03ef6d11244808c7af265f436dbf8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40449 } }
I20260924 04:13:01.575167 23071 leader_election.cc:288] T 00000000000000000000000000000000 P d1e2b25261fe4f129a81c81041c75a84 [CANDIDATE]: Term 1 election: Requested vote from peers fd728e053cd4403cb43f85661c627ada (127.0.98.61:36631), 14c03ef6d11244808c7af265f436dbf8 (127.0.98.60:40449)
I20260924 04:13:01.575647 23146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d1e2b25261fe4f129a81c81041c75a84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "fd728e053cd4403cb43f85661c627ada"
I20260924 04:13:01.575879 23146 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd728e053cd4403cb43f85661c627ada [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:13:01.577250 23146 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fd728e053cd4403cb43f85661c627ada [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e2b25261fe4f129a81c81041c75a84 in term 1.
I20260924 04:13:01.578087 23045 leader_election.cc:302] T 00000000000000000000000000000000 P d1e2b25261fe4f129a81c81041c75a84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1e2b25261fe4f129a81c81041c75a84, fd728e053cd4403cb43f85661c627ada; no voters:
I20260924 04:13:01.578357 23268 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d1e2b25261fe4f129a81c81041c75a84 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:13:01.578773 23268 raft_consensus.cc:686] T 00000000000000000000000000000000 P d1e2b25261fe4f129a81c81041c75a84 [term 1 LEADER]: Becoming Leader. State: Replica: d1e2b25261fe4f129a81c81041c75a84, State: Running, Role: LEADER
I20260924 04:13:01.579350 23268 consensus_queue.cc:237] T 00000000000000000000000000000000 P d1e2b25261fe4f129a81c81041c75a84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e2b25261fe4f129a81c81041c75a84" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33101 } } peers { permanent_uuid: "fd728e053cd4403cb43f85661c627ada" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36631 } } peers { permanent_uuid: "14c03ef6d11244808c7af265f436dbf8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40449 } }
W20260924 04:13:01.580760 23280 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d1e2b25261fe4f129a81c81041c75a84: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:13:01.580835 23280 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:13:01.582334 23268 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1e2b25261fe4f129a81c81041c75a84 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d1e2b25261fe4f129a81c81041c75a84. Latest consensus state: current_term: 1 leader_uuid: "d1e2b25261fe4f129a81c81041c75a84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e2b25261fe4f129a81c81041c75a84" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33101 } } peers { permanent_uuid: "fd728e053cd4403cb43f85661c627ada" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36631 } } peers { permanent_uuid: "14c03ef6d11244808c7af265f436dbf8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40449 } } }
I20260924 04:13:01.582549 23268 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1e2b25261fe4f129a81c81041c75a84 [sys.catalog]: This master's current role is: LEADER
I20260924 04:13:01.580893 23029 master_runner.cc:186] Error getting master registration for 127.0.98.60:40449: OK, instance_id { permanent_uuid: "14c03ef6d11244808c7af265f436dbf8" instance_seqno: 1790223181512871 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 04:13:01.584235 23029 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 04:13:01.589293 23247 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14c03ef6d11244808c7af265f436dbf8: No bootstrap required, opened a new log
I20260924 04:13:01.589569 23288 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:13:01.590519 23288 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:13:01.593611 23247 raft_consensus.cc:348] T 00000000000000000000000000000000 P 14c03ef6d11244808c7af265f436dbf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e2b25261fe4f129a81c81041c75a84" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33101 } } peers { permanent_uuid: "fd728e053cd4403cb43f85661c627ada" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36631 } } peers { permanent_uuid: "14c03ef6d11244808c7af265f436dbf8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40449 } }
I20260924 04:13:01.593931 23247 raft_consensus.cc:729] T 00000000000000000000000000000000 P 14c03ef6d11244808c7af265f436dbf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14c03ef6d11244808c7af265f436dbf8, State: Initialized, Role: FOLLOWER
I20260924 04:13:01.594327 23247 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14c03ef6d11244808c7af265f436dbf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e2b25261fe4f129a81c81041c75a84" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33101 } } peers { permanent_uuid: "fd728e053cd4403cb43f85661c627ada" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36631 } } peers { permanent_uuid: "14c03ef6d11244808c7af265f436dbf8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40449 } }
I20260924 04:13:01.595755 23247 sys_catalog.cc:564] T 00000000000000000000000000000000 P 14c03ef6d11244808c7af265f436dbf8 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:13:01.595804 23221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d1e2b25261fe4f129a81c81041c75a84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "14c03ef6d11244808c7af265f436dbf8"
I20260924 04:13:01.596558 23221 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 14c03ef6d11244808c7af265f436dbf8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:13:01.598018 23221 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 14c03ef6d11244808c7af265f436dbf8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e2b25261fe4f129a81c81041c75a84 in term 1.
I20260924 04:13:01.598061 23177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:01.585306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47790 (local address 127.0.98.61:36631)
0924 04:13:01.585462 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:01.585466 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:01.585899 (+ 433us) server_negotiation.cc:407] Connection header received
0924 04:13:01.586232 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:01.586235 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:01.586287 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:01.586343 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:13:01.588137 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:01.588874 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:01.590744 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:01.591042 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:01.591133 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:01.593495 (+ 2362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:13:01.593499 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:13:01.593502 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:13:01.593529 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:13:01.595267 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:01.596580 (+ 1313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:01.596585 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:01.596591 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:01.596712 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:01.597041 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:01.597045 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:01.597047 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:01.597275 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:13:01.597378 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 04:13:01.597734 (+ 356us) server_negotiation.cc:299] Negotiation successful
0924 04:13:01.597867 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 04:13:01.604358 23100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:01.586069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35528 (local address 127.0.98.62:33101)
0924 04:13:01.586187 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:01.586192 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:01.592330 (+ 6138us) server_negotiation.cc:407] Connection header received
0924 04:13:01.592528 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:01.592532 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:01.592582 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:01.592632 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:13:01.594144 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:01.594926 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:01.598835 (+ 3909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:01.599144 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:01.599223 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:01.600597 (+ 1374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:13:01.600600 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:13:01.600603 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:13:01.600636 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:13:01.602306 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:01.602784 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:01.602788 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:01.602789 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:01.602871 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:01.603167 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:01.603170 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:01.603172 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:01.603376 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:13:01.603520 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0924 04:13:01.604097 (+ 577us) server_negotiation.cc:299] Negotiation successful
0924 04:13:01.604214 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 04:13:01.604625 23221 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 14c03ef6d11244808c7af265f436dbf8 [term 1 FOLLOWER]: Refusing update from remote peer d1e2b25261fe4f129a81c81041c75a84: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:13:01.605311 23286 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d1e2b25261fe4f129a81c81041c75a84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14c03ef6d11244808c7af265f436dbf8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:13:01.605849 23146 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fd728e053cd4403cb43f85661c627ada [term 1 FOLLOWER]: Refusing update from remote peer d1e2b25261fe4f129a81c81041c75a84: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:13:01.609359 23286 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d1e2b25261fe4f129a81c81041c75a84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd728e053cd4403cb43f85661c627ada" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:13:01.611088 23252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:01.586781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42166 (local address 127.0.98.60:40449)
0924 04:13:01.586897 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:01.586902 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:01.593449 (+ 6547us) server_negotiation.cc:407] Connection header received
0924 04:13:01.593589 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:01.593591 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:01.593642 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:01.593722 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:13:01.595123 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:01.595817 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:01.597005 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:01.597360 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:01.597454 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:01.600826 (+ 3372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:13:01.600833 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:13:01.600838 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:13:01.600869 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:13:01.606356 (+ 5487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:01.609148 (+ 2792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:01.609151 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:01.609153 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:01.609233 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:13:01.609719 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:13:01.609725 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:13:01.609729 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:13:01.609975 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:13:01.610130 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 04:13:01.610821 (+ 691us) server_negotiation.cc:299] Negotiation successful
0924 04:13:01.610938 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 04:13:01.613572 23254 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd728e053cd4403cb43f85661c627ada [sys.catalog]: SysCatalogTable state changed. Reason: New leader d1e2b25261fe4f129a81c81041c75a84. Latest consensus state: current_term: 1 leader_uuid: "d1e2b25261fe4f129a81c81041c75a84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e2b25261fe4f129a81c81041c75a84" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33101 } } peers { permanent_uuid: "fd728e053cd4403cb43f85661c627ada" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36631 } } peers { permanent_uuid: "14c03ef6d11244808c7af265f436dbf8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40449 } } }
I20260924 04:13:01.613708 23254 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd728e053cd4403cb43f85661c627ada [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:13:01.614405 23305 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14c03ef6d11244808c7af265f436dbf8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d1e2b25261fe4f129a81c81041c75a84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e2b25261fe4f129a81c81041c75a84" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33101 } } peers { permanent_uuid: "fd728e053cd4403cb43f85661c627ada" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36631 } } peers { permanent_uuid: "14c03ef6d11244808c7af265f436dbf8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40449 } } }
I20260924 04:13:01.614593 23305 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14c03ef6d11244808c7af265f436dbf8 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:13:01.616024 23292 mvcc.cc:205] Tried to move back new op lower bound from 7332754151844601856 to 7332754151755456512. Current Snapshot: MvccSnapshot[applied={T|T < 7332754151844601856}]
I20260924 04:13:01.616153 23268 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1e2b25261fe4f129a81c81041c75a84 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d1e2b25261fe4f129a81c81041c75a84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e2b25261fe4f129a81c81041c75a84" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33101 } } peers { permanent_uuid: "fd728e053cd4403cb43f85661c627ada" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36631 } } peers { permanent_uuid: "14c03ef6d11244808c7af265f436dbf8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40449 } } }
I20260924 04:13:01.616250 23268 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1e2b25261fe4f129a81c81041c75a84 [sys.catalog]: This master's current role is: LEADER
I20260924 04:13:01.616750 23305 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14c03ef6d11244808c7af265f436dbf8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d1e2b25261fe4f129a81c81041c75a84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e2b25261fe4f129a81c81041c75a84" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33101 } } peers { permanent_uuid: "fd728e053cd4403cb43f85661c627ada" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36631 } } peers { permanent_uuid: "14c03ef6d11244808c7af265f436dbf8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40449 } } }
I20260924 04:13:01.616925 23305 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14c03ef6d11244808c7af265f436dbf8 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:13:01.617120 23306 mvcc.cc:205] Tried to move back new op lower bound from 7332754151844601856 to 7332754151755456512. Current Snapshot: MvccSnapshot[applied={T|T < 7332754151844601856}]
W20260924 04:13:01.617208 23304 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 14c03ef6d11244808c7af265f436dbf8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:13:01.617286 23304 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:13:01.617451 23293 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14c03ef6d11244808c7af265f436dbf8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d1e2b25261fe4f129a81c81041c75a84. Latest consensus state: current_term: 1 leader_uuid: "d1e2b25261fe4f129a81c81041c75a84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e2b25261fe4f129a81c81041c75a84" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33101 } } peers { permanent_uuid: "fd728e053cd4403cb43f85661c627ada" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36631 } } peers { permanent_uuid: "14c03ef6d11244808c7af265f436dbf8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40449 } } }
I20260924 04:13:01.617542 23293 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14c03ef6d11244808c7af265f436dbf8 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:13:01.619941 23268 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1e2b25261fe4f129a81c81041c75a84 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d1e2b25261fe4f129a81c81041c75a84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e2b25261fe4f129a81c81041c75a84" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33101 } } peers { permanent_uuid: "fd728e053cd4403cb43f85661c627ada" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36631 } } peers { permanent_uuid: "14c03ef6d11244808c7af265f436dbf8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40449 } } }
I20260924 04:13:01.620045 23268 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1e2b25261fe4f129a81c81041c75a84 [sys.catalog]: This master's current role is: LEADER
I20260924 04:13:01.622500 23288 catalog_manager.cc:1347] Generated new cluster ID: a6cfdefe9da740dca6c6d0a27abced7e
I20260924 04:13:01.622560 23288 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:13:01.623417 23254 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd728e053cd4403cb43f85661c627ada [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d1e2b25261fe4f129a81c81041c75a84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e2b25261fe4f129a81c81041c75a84" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33101 } } peers { permanent_uuid: "fd728e053cd4403cb43f85661c627ada" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36631 } } peers { permanent_uuid: "14c03ef6d11244808c7af265f436dbf8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40449 } } }
I20260924 04:13:01.623539 23254 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd728e053cd4403cb43f85661c627ada [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:13:01.639148 23288 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:13:01.642292 23288 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:13:01.652508 23288 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d1e2b25261fe4f129a81c81041c75a84: Generated new TSK 0
I20260924 04:13:01.653451 23288 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 04:13:01.689013 23100 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:13:01.685978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35538 (local address 127.0.98.62:33101)
0924 04:13:01.686111 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:01.686115 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:01.687584 (+ 1469us) server_negotiation.cc:407] Connection header received
0924 04:13:01.687979 (+ 395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:01.687982 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:01.688035 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:01.688103 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:13:01.688867 (+ 764us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35538: BlockingRecv error: recv got EOF from 127.0.0.1:35538 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260924 04:13:01.690231 23177 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:13:01.685752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47796 (local address 127.0.98.61:36631)
0924 04:13:01.685892 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:01.685897 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:01.687774 (+ 1877us) server_negotiation.cc:407] Connection header received
0924 04:13:01.689433 (+ 1659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:01.689436 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:01.689489 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:01.689543 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:13:01.690107 (+ 564us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47796: BlockingRecv error: recv got EOF from 127.0.0.1:47796 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260924 04:13:01.691823 23252 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:13:01.686406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42178 (local address 127.0.98.60:40449)
0924 04:13:01.686518 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:01.686523 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:01.690907 (+ 4384us) server_negotiation.cc:407] Connection header received
0924 04:13:01.691131 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:01.691134 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:01.691185 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:01.691236 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:13:01.691705 (+ 469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42178: BlockingRecv error: recv got EOF from 127.0.0.1:42178 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260924 04:13:01.696278 23177 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:13:01.694703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47798 (local address 127.0.98.61:36631)
0924 04:13:01.694800 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:01.694804 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:01.695313 (+ 509us) server_negotiation.cc:407] Connection header received
0924 04:13:01.695531 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:01.695534 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:01.695611 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:01.695662 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:13:01.696182 (+ 520us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47798: BlockingRecv error: recv got EOF from 127.0.0.1:47798 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260924 04:13:01.701427 23177 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:13:01.699948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47804 (local address 127.0.98.61:36631)
0924 04:13:01.700044 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:01.700048 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:01.700504 (+ 456us) server_negotiation.cc:407] Connection header received
0924 04:13:01.700722 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:01.700726 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:01.700778 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:01.700831 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:13:01.701324 (+ 493us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47804: BlockingRecv error: recv got EOF from 127.0.0.1:47804 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260924 04:13:01.707228 23177 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:13:01.705760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47812 (local address 127.0.98.61:36631)
0924 04:13:01.705861 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:01.705865 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:01.706341 (+ 476us) server_negotiation.cc:407] Connection header received
0924 04:13:01.706525 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:01.706529 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:01.706579 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:01.706629 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:13:01.707123 (+ 494us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47812: BlockingRecv error: recv got EOF from 127.0.0.1:47812 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260924 04:13:01.714986 23177 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:13:01.713518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47824 (local address 127.0.98.61:36631)
0924 04:13:01.713648 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:01.713652 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:01.714111 (+ 459us) server_negotiation.cc:407] Connection header received
0924 04:13:01.714301 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:01.714304 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:01.714354 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:01.714404 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:13:01.714882 (+ 478us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47824: BlockingRecv error: recv got EOF from 127.0.0.1:47824 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260924 04:13:01.726794 23177 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:13:01.725318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47834 (local address 127.0.98.61:36631)
0924 04:13:01.725423 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:01.725427 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:01.725883 (+ 456us) server_negotiation.cc:407] Connection header received
0924 04:13:01.726066 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:01.726070 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:01.726133 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:01.726183 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:13:01.726684 (+ 501us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47834: BlockingRecv error: recv got EOF from 127.0.0.1:47834 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260924 04:13:01.747110 23177 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:13:01.745466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47836 (local address 127.0.98.61:36631)
0924 04:13:01.745598 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:01.745603 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:01.746135 (+ 532us) server_negotiation.cc:407] Connection header received
0924 04:13:01.746343 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:01.746347 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:01.746401 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:01.746453 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:13:01.746981 (+ 528us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47836: BlockingRecv error: recv got EOF from 127.0.0.1:47836 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260924 04:13:01.783824 23177 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:13:01.782021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47852 (local address 127.0.98.61:36631)
0924 04:13:01.782167 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:01.782172 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:01.782752 (+ 580us) server_negotiation.cc:407] Connection header received
0924 04:13:01.782985 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:01.782989 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:01.783046 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:01.783112 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:13:01.783698 (+ 586us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47852: BlockingRecv error: recv got EOF from 127.0.0.1:47852 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260924 04:13:01.851828 23177 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:13:01.850691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47860 (local address 127.0.98.61:36631)
0924 04:13:01.850904 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:01.850910 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:01.850935 (+ 25us) server_negotiation.cc:407] Connection header received
0924 04:13:01.851065 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:01.851070 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:01.851142 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:01.851219 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:13:01.851669 (+ 450us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47860: BlockingRecv error: recv got EOF from 127.0.0.1:47860 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
W20260924 04:13:01.982852 23177 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:13:01.981768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47876 (local address 127.0.98.61:36631)
0924 04:13:01.981903 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:01.981906 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:01.982067 (+ 161us) server_negotiation.cc:407] Connection header received
0924 04:13:01.982174 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:01.982176 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:01.982223 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:01.982303 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:13:01.982712 (+ 409us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47876: BlockingRecv error: recv got EOF from 127.0.0.1:47876 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260924 04:13:02.242446 23177 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:13:02.241276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47892 (local address 127.0.98.61:36631)
0924 04:13:02.241427 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:02.241431 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:02.241450 (+ 19us) server_negotiation.cc:407] Connection header received
0924 04:13:02.241644 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:02.241648 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:02.241701 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:02.241779 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:13:02.242301 (+ 522us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47892: BlockingRecv error: recv got EOF from 127.0.0.1:47892 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 04:13:02.569794 23265 catalog_manager.cc:1259] Loaded cluster ID: a6cfdefe9da740dca6c6d0a27abced7e
I20260924 04:13:02.569872 23265 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fd728e053cd4403cb43f85661c627ada: loading cluster ID for follower catalog manager: success
I20260924 04:13:02.571470 23265 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fd728e053cd4403cb43f85661c627ada: acquiring CA information for follower catalog manager: success
I20260924 04:13:02.572576 23265 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fd728e053cd4403cb43f85661c627ada: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 04:13:02.618592 23304 catalog_manager.cc:1259] Loaded cluster ID: a6cfdefe9da740dca6c6d0a27abced7e
I20260924 04:13:02.618675 23304 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 14c03ef6d11244808c7af265f436dbf8: loading cluster ID for follower catalog manager: success
I20260924 04:13:02.620270 23304 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 14c03ef6d11244808c7af265f436dbf8: acquiring CA information for follower catalog manager: success
I20260924 04:13:02.621433 23304 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 14c03ef6d11244808c7af265f436dbf8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 04:13:02.760640 23379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:02.756606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47898 (local address 127.0.98.61:36631)
0924 04:13:02.756893 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:02.756898 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:02.756976 (+ 78us) server_negotiation.cc:407] Connection header received
0924 04:13:02.757042 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:02.757045 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:02.757183 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:02.757308 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:13:02.757581 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:02.758125 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:02.758995 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:02.759234 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:02.759366 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:02.759719 (+ 353us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:13:02.760166 (+ 447us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:13:02.760311 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0924 04:13:02.760346 (+ 35us) server_negotiation.cc:299] Negotiation successful
0924 04:13:02.760403 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":104,"threads_started":1}
I20260924 04:13:02.767542 23385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:02.763858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42186 (local address 127.0.98.60:40449)
0924 04:13:02.764090 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:02.764094 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:02.764138 (+ 44us) server_negotiation.cc:407] Connection header received
0924 04:13:02.764281 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:02.764283 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:02.764398 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:02.764496 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:13:02.764733 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:02.765255 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:02.766063 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:02.766278 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:02.766354 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:02.766693 (+ 339us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:13:02.767128 (+ 435us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:13:02.767221 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 04:13:02.767304 (+ 83us) server_negotiation.cc:299] Negotiation successful
0924 04:13:02.767349 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":54,"threads_started":1}
I20260924 04:13:02.775032 23379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:02.770717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47906 (local address 127.0.98.61:36631)
0924 04:13:02.770902 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:02.770906 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:02.771300 (+ 394us) server_negotiation.cc:407] Connection header received
0924 04:13:02.771394 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:02.771397 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:02.771478 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:02.771550 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:13:02.771850 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:02.772512 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:02.773778 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:02.774144 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:02.774237 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:02.774533 (+ 296us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:13:02.774632 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:13:02.774679 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 04:13:02.774840 (+ 161us) server_negotiation.cc:299] Negotiation successful
0924 04:13:02.774894 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260924 04:13:02.775293 23385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:02.770815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42202 (local address 127.0.98.60:40449)
0924 04:13:02.770923 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:02.770929 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:02.771505 (+ 576us) server_negotiation.cc:407] Connection header received
0924 04:13:02.771603 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:02.771608 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:02.771668 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:02.771729 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:13:02.772049 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:02.772833 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:02.773875 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:02.774261 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:02.774351 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:02.774758 (+ 407us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:13:02.774833 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:13:02.774874 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0924 04:13:02.775102 (+ 228us) server_negotiation.cc:299] Negotiation successful
0924 04:13:02.775163 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 04:13:02.775619 23391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:13:02.770718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35554 (local address 127.0.98.62:33101)
0924 04:13:02.771070 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0924 04:13:02.771074 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:13:02.771174 (+ 100us) server_negotiation.cc:407] Connection header received
0924 04:13:02.771339 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:13:02.771344 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:13:02.771523 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:13:02.771690 (+ 167us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:13:02.771964 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:02.772835 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:02.773970 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:13:02.774321 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:13:02.774434 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:13:02.774820 (+ 386us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:13:02.775173 (+ 353us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:13:02.775267 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 04:13:02.775350 (+ 83us) server_negotiation.cc:299] Negotiation successful
0924 04:13:02.775499 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":62,"threads_started":1}
I20260924 04:13:02.779794 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 23029
I20260924 04:13:02.786581 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 23104
I20260924 04:13:02.792488 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1CVu5m/build/debug/bin/kudu with pid 23178
2026-09-24T04:13:02Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3079 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6155 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (283978 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST